KVM - Guest VM without Domain ID

General support questions
Post Reply
User avatar
penguinpages
Posts: 91
Joined: 2015/07/21 13:58:05

KVM - Guest VM without Domain ID

Post by penguinpages » 2019/09/11 14:45:23

I have been using a script to deploy cluster nodes via script.

Some how.. I create the VM list... and though I used notepad+ it may have left some kind of weird character at end (this is my guess for what happened as I got weird names of disk files etc...)

What I need to do is clean this sillyness up... but with CentOS KVM.. not very clear on where these data points are saved to clear out VM definitions

###
[root@thor qemu]# virsh list --all
Id Name State
----------------------------------------------------
1 tprod running
2 tdev running
3 ns01 running
26 osworker01 running
28 osworker02 running
30 osworker03 running
32 osmstr01 running
34 osmstr02 running
36 osmstr03 running
38 osinf01 running
40 osinf02 running
42 osinf03 running
44 test001 running
46 test002 running
- centos7 shut off
- osinf01
shut off
- osmstr03
shut off

- rhel7 shut off
- undercloud00 shut off

############


I don't think this is the structure it pulls from as there were VMs here which were long ago deleted and left pieces over here...

[root@thor qemu]# pwd
/var/lib/libvirt/qemu
[root@thor qemu]# ls -alh
total 4.0K
drwxr-x--x 25 qemu qemu 4.0K Sep 11 10:33 .
drwxr-xr-x 10 root root 117 Aug 27 20:42 ..
drwxr-xr-x 3 qemu qemu 20 Feb 7 2019 channel
drwxr-x--- 2 qemu qemu 26 Jul 30 16:06 domain-16-openshift00
drwxr-x--- 2 qemu qemu 26 Aug 27 20:54 domain-1-tprod
drwxr-x--- 2 qemu qemu 26 Sep 10 15:45 domain-26-osworker01
drwxr-x--- 2 qemu qemu 26 Sep 10 15:46 domain-28-osworker02
drwxr-x--- 2 qemu qemu 26 Aug 27 20:54 domain-2-tdev
drwxr-x--- 2 qemu qemu 26 Sep 10 15:48 domain-30-osworker03
drwxr-x--- 2 qemu qemu 26 Sep 11 08:18 domain-32-osmstr01
drwxr-x--- 2 qemu qemu 26 Sep 11 08:20 domain-34-osmstr02
drwxr-x--- 2 qemu qemu 26 Sep 11 08:22 domain-36-osmstr03
drwxr-x--- 2 qemu qemu 26 Sep 11 08:23 domain-38-osinf01
drwxr-x--- 2 qemu qemu 26 Mar 23 08:16 domain-3-kmstr02
drwxr-x--- 2 qemu qemu 26 Aug 27 20:54 domain-3-ns01
drwxr-x--- 2 qemu qemu 26 Sep 11 08:25 domain-40-osinf02
drwxr-x--- 2 qemu qemu 26 Sep 11 08:27 domain-42-osinf03
drwxr-x--- 2 qemu qemu 26 Sep 11 09:05 domain-44-test001
drwxr-x--- 2 qemu qemu 26 Sep 11 09:07 domain-46-test002
drwxr-x--- 2 qemu qemu 26 Aug 12 07:34 domain-4-guestfs-xbs2xbcz3rgt
drwxr-xr-x 2 qemu qemu 6 Feb 7 2019 dump
drwxr-xr-x 2 qemu qemu 6 Feb 7 2019 nvram
drwxr-xr-x 3 qemu qemu 21 Feb 7 2019 ram
drwxr-xr-x 2 qemu qemu 6 Aug 10 08:22 save
drwxr-xr-x 3 qemu qemu 21 Sep 11 10:06 snapshot
[root@thor qemu]#

I can't find CLI means to attempt to delete at all.. but in GUI.. at least I can right click and say delete... but this is error
Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper
callback(asyncjob, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/delete.py", line 165, in _async_delete
undefine = self.vm.is_persistent()
File "/usr/share/virt-manager/virtManager/domain.py", line 488, in is_persistent
return bool(self._backend.isPersistent())
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1411, in isPersistent
if ret == -1: raise libvirtError ('virDomainIsPersistent() failed', dom=self)
libvirtError: Domain not found: no domain with matching uuid 'ed09b7bd-cff3-4a34-b1a2-55a44f285755' (osinf01
)


So question is:
1) Where are the definitions saved / pulled from to list what VMs are present?
2) Why are their no domain IDs?
3) How do I delete them?

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: KVM - Guest VM without Domain ID

Post by TrevorH » 2019/09/11 16:08:09

I suspect notepad+ being at least primarily an editor for Windows, will use DOS style line endings rather than unix ones. You probabyl need to run the file you edited through dos2unix to fix it. There's probably also an option in notepad+ to save in unix format in the first place.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

User avatar
penguinpages
Posts: 91
Joined: 2015/07/21 13:58:05

Re: KVM - Guest VM without Domain ID

Post by penguinpages » 2019/09/13 13:59:08

yes..

but.. how do I delete the VMs now. I guess my beef is no matter how fubar the name field is that it creates... it should not create VMs without domain IDs, or block you from deleting. Funky Characters are just that... but are still ASCI characters.

Maybe better question is ... where is libvirt pulling that VM list? So I can edit it manually and remove the guests

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: KVM - Guest VM without Domain ID

Post by hunter86_bg » 2019/09/15 05:43:47

Have you tried with invoking virsh and inside virsh shell to undefine them ?

Otherwise check the '/etc' or '/var/lib' directories.

Last resort -> use "grub -R 'shut off' /etc" & "grub -R 'shut off' /var/lib"

Don't forget you got the option called 'ionice' to reduce stress to the system.

User avatar
penguinpages
Posts: 91
Joined: 2015/07/21 13:58:05

Re: KVM - Guest VM without Domain ID

Post by penguinpages » 2019/09/23 17:02:52

Multiple projects so sorry for delay on post response.

I tried to find way to pass "carriage return" (as I think that may somehow the be special character appended to the end of these VMs... but no dice.

[root@odin ~]# virsh list --all | awk '{print $2}' | grep -v Name | grep -v off | grep -v centos7 | grep -v ibmspectrumdiscover |grep -v rhel7 | grep -v sethlnx01

okdetcd01
okdetcd02
okdetcd03
okdmstr01
okdmstr02
okdproxy01
okdproxy02
okdproxy03
okdworker01
okdworker02
okdworker03

[root@odin ~]# for vm in $(virsh list --all | awk '{print $2}' | grep -v Name | grep -v off | grep -v centos7 | grep -v ibmspectrumdiscover |grep -v rhel7 | grep -v sethlnx01 ); do virsh undefine $vm ; done
error: failed to get domain 'okdetcd01'
error: Domain not found: no domain with matching name 'okdetcd01'

error: failed to get domain 'okdetcd02'
error: Domain not found: no domain with matching name 'okdetcd02'

error: failed to get domain 'okdetcd03'
error: Domain not found: no domain with matching name 'okdetcd03'

error: failed to get domain 'okdmstr01'
error: Domain not found: no domain with matching name 'okdmstr01'

error: failed to get domain 'okdmstr02'
error: Domain not found: no domain with matching name 'okdmstr02'

error: failed to get domain 'okdproxy01'
error: Domain not found: no domain with matching name 'okdproxy01'

error: failed to get domain 'okdproxy02'
error: Domain not found: no domain with matching name 'okdproxy02'

error: failed to get domain 'okdproxy03'
error: Domain not found: no domain with matching name 'okdproxy03'

error: failed to get domain 'okdworker01'
error: Domain not found: no domain with matching name 'okdworker01'

error: failed to get domain 'okdworker02'
error: Domain not found: no domain with matching name 'okdworker02'

error: failed to get domain 'okdworker03'
error: Domain not found: no domain with matching name 'okdworker03'



Tried within the virsh shell
[root@odin ~]# virsh
virsh # list --all
Id Name State
----------------------------------------------------
4 centos7 running
5 ibmspectrumdiscover running
6 rhel7 running
7 sethlnx01 running
- okdetcd01
shut off
- okdetcd02
shut off
- okdetcd03
shut off
- okdmstr01
shut off
- okdmstr02
shut off
- okdproxy01
shut off
- okdproxy02
shut off
- okdproxy03
shut off
- okdworker01
shut off
- okdworker02
shut off
- okdworker03
shut off

virsh # undefine okdetcd01
error: failed to get domain 'okdetcd01'
error: Domain not found: no domain with matching name 'okdetcd01'

#

I finally got GUI back working .. VDO perl conflict with something in VirtManager...


And now that relying on GUI :P I can now clean up VMs
VirtManager_VM_Rename.png
VirtManager_VM_Rename.png (143.17 KiB) Viewed 5829 times

virsh # list --all
Id Name State
----------------------------------------------------
4 centos7 running
5 ibmspectrumdiscover running
6 rhel7 running
7 sethlnx01 running
- foo shut off
- okdetcd02
shut off
- okdetcd03
shut off
- okdmstr01
shut off
- okdmstr02
shut off
- okdproxy01
shut off
- okdproxy02
shut off
- okdproxy03
shut off
- okdworker01
shut off
- okdworker02
shut off
- okdworker03
shut off

virsh # undefine foo
Domain foo has been undefined

virsh #

BUT....

[root@odin ~]# systemctl stop libvirtd
[root@odin ~]# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; disabled; vendor preset: enabled)
Active: inactive (dead) since Mon 2019-09-23 13:05:44 EDT; 7s ago
Docs: man:libvirtd(8)
https://libvirt.org
Process: 27977 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=0/SUCCESS)
Main PID: 27977 (code=exited, status=0/SUCCESS)
Tasks: 4 (limit: 32768)
CGroup: /system.slice/libvirtd.service
├─3208 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/docker-machines.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
├─3209 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/docker-machines.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
├─3252 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
└─3253 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

Sep 23 11:17:18 odin.penguinpage.local libvirtd[27977]: 2019-09-23 15:17:18.202+0000: 27977: info : libvirt version: 4.5.0, package: 23.el7_7.1 (CentOS BuildSystem <http://bugs.centos.org>, 2019-09-13-18:01:52, x86-...sys.centos.org)
Sep 23 11:17:18 odin.penguinpage.local libvirtd[27977]: 2019-09-23 15:17:18.202+0000: 27977: info : hostname: odin.penguinpage.local
Sep 23 11:17:18 odin.penguinpage.local libvirtd[27977]: 2019-09-23 15:17:18.202+0000: 27977: error : virNetSocketReadWire:1806 : End of file while reading data: Input/output error
Sep 23 12:41:41 odin.penguinpage.local libvirtd[27977]: 2019-09-23 16:41:41.853+0000: 27977: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor
Sep 23 13:05:05 odin.penguinpage.local libvirtd[27977]: 2019-09-23 17:05:05.968+0000: 27977: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor
Sep 23 13:05:17 odin.penguinpage.local libvirtd[27977]: 2019-09-23 17:05:17.392+0000: 27977: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor
Sep 23 13:05:25 odin.penguinpage.local libvirtd[27977]: 2019-09-23 17:05:25.970+0000: 27977: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor
Sep 23 13:05:26 odin.penguinpage.local libvirtd[27977]: 2019-09-23 17:05:26.130+0000: 27977: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor
Sep 23 13:05:44 odin.penguinpage.local systemd[1]: Stopping Virtualization daemon...
Sep 23 13:05:44 odin.penguinpage.local systemd[1]: Stopped Virtualization daemon.
Hint: Some lines were ellipsized, use -l to show in full.
[root@odin ~]# systemctl start libvirtd
[root@odin ~]#


THERE BAAACK!!!! <insert creepy voice over>

[root@odin ~]# virsh list --all
Id Name State
----------------------------------------------------
- centos7 shut off
- ibmspectrumdiscover shut off
- okdetcd01
shut off
- okdetcd02
shut off
- okdetcd03
shut off
- okdmstr01
shut off
- okdmstr02
shut off
- okdproxy01
shut off
- okdproxy02
shut off
- okdproxy03
shut off
- okdworker01
shut off
- okdworker02
shut off
- okdworker03
shut off
- rhel7 shut off
- sethlnx01 shut off

[root@odin ~]#


Maybe I should re post this as the undead VMs.... they can't be killed!!

User avatar
penguinpages
Posts: 91
Joined: 2015/07/21 13:58:05

Re: KVM - Guest VM without Domain ID

Post by penguinpages » 2019/10/02 13:16:04

This is defiantly something wrong with my script to deploy groups of VMs for a cluster.

New server, fresh gluster file space. Fresh KVM environment. Create template RHEL7 VM. Check my VM list to validate no "weird characters"....and now I have ghost VMs.. Symptom I get is when it starts to create the VM.. it fails on attaching the "opt" and "var" volumes to system as it lists a space between hostname and volume... so it fails to attach the the VM

[root@medusa ~]# virsh list --all
Id Name State
----------------------------------------------------
- osmstr01
shut off
- rhel7 shut off

[root@medusa ~]#
vm_weird_character.png
vm_weird_character.png (73.5 KiB) Viewed 5771 times

Attempt to create issue again for baseline


# List of VMs in cluster to create
[root@medusa ~]# cat /media/sw2_usb_A1/penguinpages_local_cluster/os_cluster_nodes
osmstr01
osmstr02
osmstr03
osinf01
osinf02
osinf03
osworker01
osworker02
osworker03
osetcd01
osetcd02
osetcd03

# check for weird characters
[root@medusa ~]# vi /media/sw2_usb_A1/penguinpages_local_cluster/os_cluster_nodes
osmstr01$
osmstr02$
osmstr03$
osinf01$
osinf02$
osinf03$
osworker01$
osworker02$
osworker03$
osetcd01$
osetcd02$
osetcd03$
~
~
:set list

# Look at deployment script to see if something weird

[root@medusa ~]# vi /media/sw2_usb_A1/penguinpages_local_cluster/vm_deploy_os
#!/bin/bash$
# read in file and run loop against list of VMs to create VM$
# v0.4 2019-10-01$
for vm in $(cat /media/sw2_usb_A1/penguinpages_local_cluster/os_cluster_nodes)$
do$
virt-clone --original rhel7 --name "$vm" --auto-clone$
virsh setvcpus "$vm" 4 --config --maximum$
virsh setvcpus "$vm" 4 --config$
virsh setmaxmem "$vm" 16G --config$
sleep 10$
$
#Create OPT and VAR volumes for servers and mount and move data$
qemu-img create -f qcow2 /data/gv0/vms/"$vm"_opt.qcow2 80G$
virsh attach-disk --persistent --subdriver qcow2 $vm /data/gv0/vms/"$vm"_opt.qcow2 vdb$
sleep 5$
qemu-img create -f qcow2 /data/gv0/vms/"$vm"_var.qcow2 100G$
virsh attach-disk --persistent --subdriver qcow2 $vm /data/gv0/vms/"$vm"_var.qcow2 vdc$
sleep 5$
virt-sysprep -d $vm --hostname "$vm" --root-password password:1q2w3e4r --firstboot /media/sw2_usb_A1/penguinpages_local_cluster/vm_post_deploy_rhel$
virsh start $vm$
$
done $
~
~
:set list


# Nothing odd.. so lets deploy a bunch of VMs!!
[root@medusa ~]# sh /media/sw2_usb_A1/penguinpages_local_cluster/vm_deploy_os
-1.qcow2' | 30 GB 00:00:17

' created successfully.



_opt.qcow2', fmt=qcow2 size=85899345920 cluster_size=65536 lazy_refcounts=off refcount_bits=16
Disk attached successfully

_var.qcow2', fmt=qcow2 size=107374182400 cluster_size=65536 lazy_refcounts=off refcount_bits=16
Disk attached successfully

[ 0.0] Examining the guest ...
virt-sysprep: error: libguestfs error: /data/gv0/vms/osmstr01 -1.qcow2: No
such file or directory

If reporting bugs, run virt-sysprep with debugging enabled and include the
complete output:

virt-sysprep -v -x [...]
error: Failed to start domain osmstr01

error: Cannot access storage file '/data/gv0/vms/osmstr01 _var.qcow2' (as uid:107, gid:107): No such file or directory

-1.qcow2' 8% [======== ] 306 MB/s | 2.7 GB 00:01:31 ETA
^C.qcow2' 10% [========= ] 312 MB/s | 3.0 GB 00:01:28 ETA
^C
^C.qcow2' 18% [================- ] 338 MB/s | 5.4 GB 00:01:14 ETA
Installation aborted at user request 21% [===================- ] 327 MB/s | 6.4 GB 00:01:14 ETA



^C
You have new mail in /var/spool/mail/root
[root@medusa ~]#


# :(

# List of VMs lies like a rug until it is restart!
[root@medusa ~]# virsh list --all
Id Name State
----------------------------------------------------
- osmstr01
shut off
- osmstr01
shut off
shut off
- rhel7 shut off

[root@medusa ~]# !systemc
systemctl restart libvirtd
[root@medusa ~]# virsh list --all
Id Name State
----------------------------------------------------
- osmstr01
shut off
- osmstr02
shut off
- rhel7 shut off

[root@medusa ~]#

# Now I have more VMs with weird characters
vm_weird_character.png
vm_weird_character.png (73.5 KiB) Viewed 5771 times

# And I cannot delete them at CLI
[root@medusa ~]# virsh list --all
Id Name State
----------------------------------------------------
- osmstr01
shut off
- osmstr02
shut off
- rhel7 shut off

[root@medusa ~]# virsh undefine osmstr01
error: failed to get domain 'osmstr01'
error: Domain not found: no domain with matching name 'osmstr01'

[root@medusa ~]# virsh undefine osmstr02
error: failed to get domain 'osmstr02'
error: Domain not found: no domain with matching name 'osmstr02'

[root@medusa ~]#

# I can delete them in GUI... or rename them and do it at CLI
delete_vm_weird_character.jpg
delete_vm_weird_character.jpg (190.53 KiB) Viewed 5771 times
[root@medusa ~]# virsh list --all
Id Name State
----------------------------------------------------
- foo shut off
- rhel7 shut off

[root@medusa ~]# virsh undefine foo
Domain foo has been undefined

[root@medusa ~]# virsh list --all
Id Name State
----------------------------------------------------
- rhel7 shut off

[root@medusa ~]# !system
systemctl restart libvirtd
[root@medusa ~]# virsh list --all
Id Name State
----------------------------------------------------
- osmstr01
shut off
- osmstr02
shut off
- rhel7 shut off

[root@medusa ~]#


Where is KVM storing this and why are they continuing to come back!! Ugh :P
Attachments
more_vm_weird_character.png
more_vm_weird_character.png (237.8 KiB) Viewed 5771 times

tony_down_under
Posts: 83
Joined: 2019/08/07 01:50:24
Location: Perth, Australia but originally from Carshalton, Surrey

Re: KVM - Guest VM without Domain ID

Post by tony_down_under » 2019/10/03 01:15:51

In notepad++ you can set the txt file to unix style by going to
Edit > EOL Conversion > Unix
I done this recently to fix a bash script I wrote on my Windows 10 system.

User avatar
penguinpages
Posts: 91
Joined: 2015/07/21 13:58:05

Re: KVM - Guest VM without Domain ID

Post by penguinpages » 2019/10/11 12:27:51

Just to close this out. Thanks to all for response.

1) "vi" which I have trusted for years and thought was the way to show this kind of char mis-configuration with "set list" is not of value. Note to self...d on't trust it
2) Notepad plus with VIEW-> Show Symbols -> Show All Characters this displayed difference: Broken = "CR LF" as end of line vs Unix "LF"
3) Once VMs are deployed libvirt gets confused and renaming name field does not fix it. only way is to shut down libvirt and wipe out the XML files for the VMs

Ex:
[root@thor home]# systemctl stop libvirtd
[root@thor home]# cd /etc/libvirt/qemu
[root@thor qemu]# ls
autostart networks osetcd01.xml osetcd03.xml osinf02.xml osmstr01.xml osmstr03.xml osworker02.xml rhel7.xml
centos7.xml ns01.xml osetcd02.xml osinf01.xml osinf03.xml osmstr02.xml osworker01.xml osworker03.xml
[root@thor qemu]# rm -rf ns01.xml
[root@thor home]# systemctl start libvirtd

I had to use wild cards for VM names as it had garbage in file name but all began with os*

Post Reply