qemu-kvm-ev issues creating a VM

[smingoli]

I'm having an issue with qemu-kvm-ev, basically it looks like I'm not able to create any VM. Everything works fine if I use the standard qemu-kvm

I have a VMWare Workstation 15.1 VM based on Centos 7.6 (Kenel 3.10.0-957) with the CentOS-QEMU-EV repository enabled.

Version of qemu is

Code: Select all

[root@localhost ~]# yum list installed |grep qemu-
centos-release-qemu-ev.noarch              1.0-4.el7.centos            @extras
libcacard.x86_64                           40:2.5.2-2.1.el7            @centos-qemu-ev-local
qemu-img-ev.x86_64                         10:2.12.0-18.el7_6.5.1      @centos-qemu-ev-local
qemu-kvm-common-ev.x86_64                  10:2.12.0-18.el7_6.5.1      @centos-qemu-ev-local
qemu-kvm-ev.x86_64                         10:2.12.0-18.el7_6.5.1      @centos-qemu-ev-local

When I try and create a KVM VM using the following command:

Code: Select all

virt-install --name=node1 \
--memory=2048 --vcpus=1 \
--location=/var/lib/libvirt/boot/Centos76.iso \
--disk /var/lib/libvirt/images/centos76.qcow2,size=10,bus=virtio,format=qcow2 \
--network=bridge=br0,model=virtio \
--os-type=linux  \
--nographics \
--extra-args='console=tty0 console=ttyS0,115200n8 serial'

I get a "NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:0H:5]" and the installation will not proceed.

Extract of the log

Code: Select all

[   33.048424] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:0H:5]
[   33.053109] Modules linked in: crct10dif_pclmul crct10dif_common 8021q(+) crc32_pclmul sr_mod(+) crc32c_intel cdrom garp mrp stp llc virtio_blk(+) ghash_clmulni_intel virtio_net ata_generic pata_acpi aesni_intel glue_helper ablk_helper cryptd serio_raw ata_piix libata virtio_pci virtio_ring virtio sunrpc xts lrw gf128mul dm_crypt dm_round_robin dm_multipath dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_zero dm_mod linear raid10 raid456 async_raid6_recov async_memcpy async_pq raid6_pq libcrc32c async_xor xor async_tx raid1 raid0 iscsi_ibft iscsi_boot_sysfs floppy iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi squashfs cramfs edd
[   33.117004] CPU: 0 PID: 5 Comm: kworker/0:0H Not tainted 3.10.0-957.el7.x86_64 #1
[   33.123024] Hardware name: Red Hat KVM, BIOS 1.11.0-2.el7 04/01/2014
[   33.128023] Workqueue: kblockd blk_mq_run_work_fn
[   33.131139] task: ffff95a25b4f4100 ti: ffff95a25b520000 task.ti: ffff95a25b520000
[   33.136155] RIP: 0010:[<ffffffff97f904f8>]  [<ffffffff97f904f8>] iowrite16+0x38/0x40
[   33.144655] RSP: 0018:ffff95a25b523c48  EFLAGS: 00010292
[   33.148833] RAX: ffffffffc054ab60 RBX: 0000000000000286 RCX: ffff95a25f78d940
[   33.153953] RDX: ffffad434013c000 RSI: ffffad434013c000 RDI: 0000000000000000
[   33.158652] RBP: ffff95a25b523c50 R08: 0000000000000001 R09: ffffffffc05380b4
[   33.162821] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff97ca9aad
[   33.167415] R13: ffff95a25b523bc8 R14: 0000000000000000 R15: 0000000000000000
[   33.172000] FS:  0000000000000000(0000) GS:ffff95a25ca00000(0000) knlGS:0000000000000000
[   33.177042] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   33.180732] CR2: 00007fee4310d4a0 CR3: 000000001f9fc000 CR4: 00000000003606f0
[   33.186887] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   33.192527] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   33.198503] Call Trace:
[   33.200616]  [<ffffffffc054ab76>] ? vp_notify+0x16/0x20 [virtio_pci]
[   33.205650]  [<ffffffffc053707c>] virtqueue_notify+0x1c/0x40 [virtio_ring]
[   33.211549]  [<ffffffffc054385e>] virtio_queue_rq+0x29e/0x2b0 [virtio_blk]
[   33.216615]  [<ffffffff97f50708>] blk_mq_dispatch_rq_list+0x268/0x620
[   33.221555]  [<ffffffff97f3d96a>] ? elv_rb_del+0x2a/0x40
[   33.226052]  [<ffffffff97f5557e>] blk_mq_do_dispatch_sched+0x7e/0x130
[   33.230740]  [<ffffffff97f55f3e>] blk_mq_sched_dispatch_requests+0x11e/0x1c0
[   33.237729]  [<ffffffff97f4e3b0>] __blk_mq_run_hw_queue+0x50/0xc0
[   33.242046]  [<ffffffff97f4e595>] blk_mq_run_work_fn+0x15/0x20
[   33.245714]  [<ffffffff97cb9d4f>] process_one_work+0x17f/0x440
[   33.250456]  [<ffffffff97cbade6>] worker_thread+0x126/0x3c0
[   33.255216]  [<ffffffff97cbacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[   33.260895]  [<ffffffff97cc1c31>] kthread+0xd1/0xe0
[   33.265362]  [<ffffffff97cc1b60>] ? insert_kthread_work+0x40/0x40
[   33.271676]  [<ffffffff98374c37>] ret_from_fork_nospec_begin+0x21/0x21
[   33.277531]  [<ffffffff97cc1b60>] ? insert_kthread_work+0x40/0x40
[   33.282217] Code: 81 fe 00 00 01 00 76 0b 0f b7 d6 89 f8 66 ef c3 0f 1f 00 55 48 c7 c6 51 40 6a 98 48 89 d7 48 89 e5 e8 5d fe ff ff 5d c3 0f 1f 00 <66> 89 3e c3 0f 1f 40 00 48 81 fe ff ff 03 00 48 89 f2 77 2c 48
[   61.048424] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:0H:5]
[   61.053218] Modules linked in: crct10dif_pclmul crct10dif_common 8021q(+) crc32_pclmul sr_mod(+) crc32c_intel cdrom garp mrp stp llc virtio_blk(+) ghash_clmulni_intel virtio_net ata_generic pata_acpi aesni_intel glue_helper ablk_helper cryptd serio_raw ata_piix libata virtio_pci virtio_ring virtio sunrpc xts lrw gf128mul dm_crypt dm_round_robin dm_multipath dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_zero dm_mod linear raid10 raid456 async_raid6_recov async_memcpy async_pq raid6_pq libcrc32c async_xor xor async_tx raid1 raid0 iscsi_ibft iscsi_boot_sysfs floppy iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi squashfs cramfs edd
[   61.111485] CPU: 0 PID: 5 Comm: kworker/0:0H Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
[   61.118810] Hardware name: Red Hat KVM, BIOS 1.11.0-2.el7 04/01/2014
[   61.124235] Workqueue: kblockd blk_mq_run_work_fn
[   61.127445] task: ffff95a25b4f4100 ti: ffff95a25b520000 task.ti: ffff95a25b520000
[   61.132421] RIP: 0010:[<ffffffff97f904f8>]  [<ffffffff97f904f8>] iowrite16+0x38/0x40
[   61.138377] RSP: 0018:ffff95a25b523c48  EFLAGS: 00010292
[   61.142526] RAX: ffffffffc054ab60 RBX: 0000000000000286 RCX: ffff95a25f78d940
[   61.148311] RDX: ffffad434013c000 RSI: ffffad434013c000 RDI: 0000000000000000
[   61.154537] RBP: ffff95a25b523c50 R08: 0000000000000001 R09: ffffffffc05380b4
[   61.159501] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff97ca9aad
[   61.165046] R13: ffff95a25b523bc8 R14: 0000000000000000 R15: 0000000000000000
[   61.171342] FS:  0000000000000000(0000) GS:ffff95a25ca00000(0000) knlGS:0000000000000000
[   61.177780] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   61.181975] CR2: 00007fee4310d4a0 CR3: 000000001f9fc000 CR4: 00000000003606f0
[   61.187409] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   61.194790] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   61.199668] Call Trace:
[   61.201740]  [<ffffffffc054ab76>] ? vp_notify+0x16/0x20 [virtio_pci]
[   61.205529]  [<ffffffffc053707c>] virtqueue_notify+0x1c/0x40 [virtio_ring]
[   61.210664]  [<ffffffffc054385e>] virtio_queue_rq+0x29e/0x2b0 [virtio_blk]
[   61.217244]  [<ffffffff97f50708>] blk_mq_dispatch_rq_list+0x268/0x620
[   61.222563]  [<ffffffff97f3d96a>] ? elv_rb_del+0x2a/0x40
[   61.227240]  [<ffffffff97f5557e>] blk_mq_do_dispatch_sched+0x7e/0x130
[   61.232699]  [<ffffffff97f55f3e>] blk_mq_sched_dispatch_requests+0x11e/0x1c0
[   61.238073]  [<ffffffff97f4e3b0>] __blk_mq_run_hw_queue+0x50/0xc0
[   61.242643]  [<ffffffff97f4e595>] blk_mq_run_work_fn+0x15/0x20
[   61.247504]  [<ffffffff97cb9d4f>] process_one_work+0x17f/0x440
[   61.251110]  [<ffffffff97cbade6>] worker_thread+0x126/0x3c0
[   61.254811]  [<ffffffff97cbacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[   61.258709]  [<ffffffff97cc1c31>] kthread+0xd1/0xe0
[   61.262579]  [<ffffffff97cc1b60>] ? insert_kthread_work+0x40/0x40
[   61.267186]  [<ffffffff98374c37>] ret_from_fork_nospec_begin+0x21/0x21
[   61.272029]  [<ffffffff97cc1b60>] ? insert_kthread_work+0x40/0x40
[   61.276598] Code: 81 fe 00 00 01 00 76 0b 0f b7 d6 89 f8 66 ef c3 0f 1f 00 55 48 c7 c6 51 40 6a 98 48 89 d7 48 89 e5 e8 5d fe ff ff 5d c3 0f 1f 00 <66> 89 3e c3 0f 1f 40 00 48 81 fe ff ff 03 00 48 89 f2 77 2c 48
[   68.357449] INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 0, t=60002 jiffies, g=4083, c=4082, q=77)
[   68.366595] All QSes seen, last rcu_sched kthread activity 60002 (4294734829-4294674827), jiffies_till_next_fqs=3
[   68.373681] kworker/0:0H    R  running task        0     5      2 0x00000008
[   68.379293] Workqueue: kblockd blk_mq_run_work_fn
[   68.383390] Call Trace:
[   68.385990]  <IRQ>  [<ffffffff97cd5d88>] sched_show_task+0xa8/0x110
[   68.392285]  [<ffffffff97d55fde>] rcu_check_callbacks+0x72e/0x730
[   68.398268]  [<ffffffff97d0ad60>] ? tick_sched_do_timer+0x50/0x50
[   68.402476]  [<ffffffff97cab136>] update_process_times+0x46/0x80
[   68.406965]  [<ffffffff97d0aad0>] tick_sched_handle+0x30/0x70
[   68.410768]  [<ffffffff97d0ad99>] tick_sched_timer+0x39/0x80
[   68.414450]  [<ffffffff97cc5f93>] __hrtimer_run_queues+0xf3/0x270
[   68.418404]  [<ffffffff97cc651f>] hrtimer_interrupt+0xaf/0x1d0
[   68.423124]  [<ffffffff97c5a2cb>] local_apic_timer_interrupt+0x3b/0x60
[   68.428114]  [<ffffffff983796c3>] smp_apic_timer_interrupt+0x43/0x60
[   68.432263]  [<ffffffff98375df2>] apic_timer_interrupt+0x162/0x170
[   68.436952]  <EOI>  [<ffffffffc05380b4>] ? virtqueue_add+0x1c4/0x4d0 [virtio_ring]
[   68.443915]  [<ffffffffc054ab60>] ? vp_synchronize_vectors+0x70/0x70 [virtio_pci]
[   68.450051]  [<ffffffff97f904f8>] ? iowrite16+0x38/0x40
[   68.477584]  [<ffffffffc054ab76>] ? vp_notify+0x16/0x20 [virtio_pci]
[   68.485559]  [<ffffffffc053707c>] virtqueue_notify+0x1c/0x40 [virtio_ring]
[   68.493911]  [<ffffffffc054385e>] virtio_queue_rq+0x29e/0x2b0 [virtio_blk]
[   68.501063]  [<ffffffff97f50708>] blk_mq_dispatch_rq_list+0x268/0x620
[   68.507818]  [<ffffffff97f3d96a>] ? elv_rb_del+0x2a/0x40
[   68.513800]  [<ffffffff97f5557e>] blk_mq_do_dispatch_sched+0x7e/0x130
[   68.523127]  [<ffffffff97f55f3e>] blk_mq_sched_dispatch_requests+0x11e/0x1c0
[   68.534202]  [<ffffffff97f4e3b0>] __blk_mq_run_hw_queue+0x50/0xc0
[   68.542580]  [<ffffffff97f4e595>] blk_mq_run_work_fn+0x15/0x20
[   68.551603]  [<ffffffff97cb9d4f>] process_one_work+0x17f/0x440
[   68.560758]  [<ffffffff97cbade6>] worker_thread+0x126/0x3c0
[   68.568731]  [<ffffffff97cbacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[   68.576418]  [<ffffffff97cc1c31>] kthread+0xd1/0xe0
[   68.581379]  [<ffffffff97cc1b60>] ? insert_kthread_work+0x40/0x40
[   68.588108]  [<ffffffff98374c37>] ret_from_fork_nospec_begin+0x21/0x21
[   68.595461]  [<ffffffff97cc1b60>] ? insert_kthread_work+0x40/0x40

Any ideas?

[TrevorH]

That crash is from the first 7.6 kernel and there have been numerous updates to it since October, run `yum update`

[smingoli]

Thanks for the reply, but I checked and I'am already on the latest 7.6 kernel (aka Kenel 3.10.0-957). As I said it works perfectly if I use the standard qemu-kvm package. The issue is only there if I use qemu-kvm-ev from the "centos-release-qemu-ev" repository.

[TrevorH]

No, you are not on the latest kernel. You're on 3.10.0-957.el7.x86_64 and the latest is currently 3.10.0-957.21.3.el7.x86_64.

[smingoli]

When you talk about kernel you mean the kernel version on the hypervisor (kvm), right?

version is now this:

Code: Select all

[root@kvm ~]# uname -a
Linux kvm 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

But I'm still having the same issue.

I've tried creating VMs with a bunch of different versions of Centos (7.3,7.4,7.6), Fedora, RHEL as Guest OS but they all fail miserably.

[TrevorH]

If it only does this when using the qemu-kvm-ev packages then I would suggest mailing to the centos-devel mailing list to report that. I don't think there is a category for it on bugs.centos.org though there may be and that would also be a good place to report this if so.

[smingoli]

Thanks, will do so.