17.1. A live migration example

17.1. A live migration example

Below is an example of how to setup a simple environment for live migration. This configuration is using NFS for the shared storage. NFS is suitable for demonstration environments but for a production environment a more robust shared storage configuration using Fibre Channel or iSCSI and GFS is recommended.

The configuration below consists of two servers (et-virt07 and et-virt08), both of them are using eth1 as their default network interface hence they are using xenbr1 as their Red Hat Virtualization networking bridge. We are using a locally attached SCSI disk (/dev/sdb) on et-virt07 for shared storage using NFS.

Setup for live migration

Create and mount the directory used for the migration:

# mkdir /xentest
# mount /dev/sdb /xentest

Important

Ensure the directory is exported with the correct options. If you are exporting the default directory /var/lib/xen/images/ make sure you only export /var/lib/xen/images/ and not/var/lib/xen/ as this directory is used by the xend daemon and other Xen components. Sharing /var/lib/xen/ will cause unpredictable behavior.

# cat /etc/exports
/xentest  *(rw,async,no_root_squash)

Verify it is exported via NFS:

# showmount -e et-virt07
Export list for et-virt07:
/xentest *

Install the virtual machine

The install command in the example used for installing the virtual machine:

# virt-install -p -f /xentest/xentestTravelVM01.dsk -s 5 -n\
xentesttravelvm01 --vnc -r 1024 -l http://porkchop.devel.redhat.com/rel-eng/RHEL5-\
Server-20070105.0/4.92/x86_64/os/ -b xenbr1

Verify environment for migration

Make sure the Xen networking bridges are configured correctly and have the same name on both hosts:

[et-virt08 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr1          8000.feffffffffff       no              peth1
vif0.1
[et-virt07 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr1          8000.feffffffffff       no              peth1
vif0.1

Check the relocation parameters have been configured in the Xen config file on both hosts:

[et-virt07 ~]# grep xend-relocation /etc/xen/xend-config.sxp |grep -v '#'
(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address '')
(xend-relocation-hosts-allow '')
[et-virt08 ~]# grep xend-relocation /etc/xen/xend-config.sxp |grep -v '#'
(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address '')
(xend-relocation-hosts-allow '')

Make sure the Xen relocation server has started and is listening on the dedicated port for Xen migrations (8002):

[et-virt07 ~]# lsof -i :8002
COMMAND  PID  USER   FD   TYPE  DEVICE SIZE NODE NAME
python 3445 root 14u IPv4 10223 TCP *:teradataordbms (LISTEN)
[et-virt08 ~]# lsof -i :8002
COMMAND  PID USER   FD   TYPE DEVICE SIZE NODE NAME
python 3252 root 14u IPv4 10901 TCP *:teradataordbms (LISTEN)

Verify the NFS directory has been mounted on the other host and you can see and access the virtual machine image and file system:

[et-virt08 ~]# df /xentest
Filesystem           1K-blocks      Used Available Use% Mounted on
et-virt07:/xentest    70562400   2379712  64598336   4% /xentest
[et-virt08 ~]# file /xentest/xentestTravelVM01.dsk 
/xentest/xentestTravelVM01.dsk: x86 boot sector; partition 1: ID=0x83,
active, starthead 1, startsector 63, 208782 sectors; partition 2: ID=0x8e, 
starthead 0, startsector 208845, 10265535 sectors, code offset 0x48
[et-virt08 ~]# touch /xentest/foo
[et-virt08 ~]# rm -f /xentest/foo

Verification of save and restore on local host

Start up the virtual machine (if it has not yet):

[et-virt07 ~]# xm li
Name             ID Mem(MiB) VCPUs State   Time(s)
Domain-0         0     1880  8     r-----  50.7
[et-virt07 ~]# xm create xentesttravelvm01
Using config file "/etc/xen/xentesttravelvm01".
Going to boot Red Hat Enterprise Linux Server (2.6.18-1.2961.el5xen)
kernel: /vmlinuz-2.6.18-1.2961.el5xen
initrd: /initrd-2.6.18-1.2961.el5xen.img
Started domain xentesttravelvm01

Verify the virtual machine is running:

[et-virt07 ~]# xm li
Name                     ID Mem(MiB) VCPUs State   Time(s)
Domain-0                 0  983      8     r-----  58.2
xentesttravelvm01        1  1024     1     -b----  9.2

Save the virtual machine on the local host:

[et-virt07 xentest]# time xm save xentesttravelvm01 xentesttravelvm01.sav
real    0m15.744s
user    0m0.188s
sys     0m0.044s
[et-virt07 xentest]# ls -lrt xentesttravelvm01.sav
-rwxr-xr-x 1 root root 1075657716 Jan 12 06:46 xentesttravelvm01.sav
[et-virt07 xentest]# xm li
Name                  ID Mem(MiB) VCPUs State   Time(s)
Domain-0              0  975      8     r-----  110.7

Restore the virtual machine on the local host:

[et-virt07 xentest]# time xm restore xentesttravelvm01.sav
real    0m12.443s
user    0m0.072s
sys     0m0.044s
[et-virt07 xentest]# xm li
Name                   ID Mem(MiB) VCPUs State   Time(s)
Domain-0               0  975      8     r-----  118.5
xentesttravelvm01      3  1023     1     -b----  0.0

Live migration test

Run a simple loop inside the guest to print out time and hostname every 3 seconds.

Note

The local host's clock is set to a different time 4hrs ahead of the remote host's clock.

# while true
> do
> hostname ; date
> sleep 3
> done
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:50:16 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:50:19 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:50:22 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:50:25 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:22:24 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:22:27 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:22:30 EST 2007

Verify the virtual machine is running:

[et-virt08 xen]# xm li
Name                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                  0  975      4     r-----  45.9
xentesttravelvm01         1  1023     1     -b----  1.3

Initiate the live migration to et-virt08. in the example below et-virt07 is the hostname you are migrating to and <domain-id> muyst be replaced with a guest domain available to the host system.

[et-virt08 ~]# xm migrate --live <domain-id> et-virt07

Verify the virtual machine has been shut down on et-virt07

[et-virt07 xentest]# xm li
Name                        ID Mem(MiB) VCPUs State   Time(s)
Domain-0                    0  975      8     r-----  161.1

Verify the virtual machine has been migrated to et-virt08:

[et-virt08 ~]# xm li
Name                        ID Mem(MiB) VCPUs State   Time(s)
Domain-0                    0  975      4     r-----  46.3
xentesttravelvm01           1  1023     1     -b----  1.6

Testing the progress and initiating the live migration

Create the following script inside the virtual machine to log date and hostname during the migration. This script performs I/O tasks on the virtual machine's file system.

#!/bin/bash

while true
do
touch /var/tmp/$$.log
echo `hostname` >>  /var/tmp/$$.log
	echo `date`     >>  /var/tmp/$$.log
	cat  /var/tmp/$$.log
	df /var/tmp
	ls -l  /var/tmp/$$.log
	sleep 3
	done

Remember, that script is only for testing purposes and unnecessary for a live migration in a production environment.

Verify the virtual machine is running on et-virt08 before we try to migrate it to et-virt07:

[et-virt08 ~]# xm li
Name                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                  0  975      4     r-----  46.3
xentesttravelvm01         1  1023     1     -b----  1.6

Initiate a live migration to et-virt07. You can add the time command to see how long the migration takes:

[et-virt08 ~]# time xm migrate --live xentesttravelvm01 et-virt07
real    0m10.378s
user    0m0.068s
sys     0m0.052s

run the script inside the guest:

# ./doit
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 62 Jan 12 02:26 /var/tmp/2279.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:30 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 124 Jan 12 02:26 /var/tmp/2279.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:30 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:33 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 186 Jan 12 02:26 /var/tmp/2279.log
Fri Jan 12 02:26:45 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:48 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:51 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:54:57 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:55:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:55:03 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 744 Jan 12 06:55 /var/tmp/2279.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007

Verify the virtual machine has been shut down on et-virt08:

[et-virt08 ~]# xm li
Name                    ID Mem(MiB) VCPUs State   Time(s)
Domain-0                0  975      4     r-----  56.3

Verify the virtual machine has started up on et-virt07:

[et-virt07 xentest]# xm li
Name                    ID Mem(MiB) VCPUs State   Time(s)
Domain-0                0  975      8     r-----  198.1
xentesttravelvm01       4  1023     1     -b----  1.0

Run through another cycle migrating from et-virt07 to et-virt08. Initiate a migration from et-virt07 to et-virt08:

[et-virt07 xentest]# time xm migrate --live xentesttravelvm01 et-virt08
real    0m11.490s
user    0m0.084s
sys     0m0.028s

Verify the virtual machine has been shut down:

[et-virt07 xentest]# xm li
Name                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                  0  975      8     r-----  221.7

Before initiating the migration start the simple script in the guest and note the change in time when migrating the guest:

# ./doit
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 62 Jan 12 06:57 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 124 Jan 12 06:57 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 186 Jan 12 06:57 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:00 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 248 Jan 12 02:30 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:03 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 310 Jan 12 02:30 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:03 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:06 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 372 Jan 12 02:30 /var/tmp/2418.log

After the migration command completes on et-virt07 verify on et-virt08 that the virtual machine has started:

[et-virt08 ~]# xm li
Name                       ID Mem(MiB) VCPUs State   Time(s)
Domain-0                   0  975      4     r-----  67.3
xentesttravelvm01          2  1023     1     -b----  0.4

and run another cycle:

[et-virt08 ~]# time xm migrate --live xentesttravelvm01 et-virt07
real    0m10.378s
user    0m0.068s
sys     0m0.052s

At this point you have successfully performed an offline and a live migration test.


Note: This documentation is provided {and copyrighted} by Red Hat®, Inc. and is released via the Open Publication License. The copyright holder has added the further requirement that Distribution of substantively modified versions of this document is prohibited without the explicit permission of the copyright holder. The CentOS project redistributes these original works (in their unmodified form) as a reference for CentOS-5 because CentOS-5 is built from publicly available, open source SRPMS. The documentation is unmodified to be compliant with upstream distribution policy. Neither CentOS-5 nor the CentOS Project are in any way affiliated with or sponsored by Red Hat®, Inc.