Migrate virtual machines using LVM without access to hypervisor

I decided to migrate a few virtual machines from one hosting service to another. Most other howtos either recommend a reinstall or need access to the hypervisor (KVM/libvirt in this case – I could use native migrate functionality), which I did not have. Of course, you need to make sure you have the same architecture (x86_64 in my case) and it is probably wise if both source and destination run on the same virtualizator, so the devices are correct.

I was using Centos 7.6 virtual machines. The target machine was booted into System Rescue CD, selected from the operator’s dashboard. The exact steps will be different if you use different OS.

Let’s start. On source and target, make sure there are a few utilities needed. The yum command is not required on target if you run System Rescue CD

yum install pv pigz nmap-netcat rubygems
gem install lvmsync

On System Rescue CD, you need to either add lvmsync to PATH or link it to /usr/local/bin. This is a one-time operating system, so don’t get too cozy, everything will be gone after reboot, so no need to do things “nicely”.

The plan

On the source machine, I use LVM, in one case also with LUKS and cryptsetup (hosted on LVM volume). We will recreate the same structure on the target machine. My particular structure is default Centos7. My virtual hard drive is /dev/vda (if it is not on target, check that you are using virtio, otherwise the machine will be slow). /dev/vda1 is ext4 boot partition, /dev/vda2 is LVM volume.

Since the copying can be slow, I will copy the boot partition, create a LVM snapshot of any other volumes, transfer the snapshot while the source machine is doing it’s job, then use lvmsync to sync the changes, thus minimizing downtime. Then I setup the bootloader, change network settings and optionally do a portforward on the old machine, to compensate for slow DNS propagation.

Recreate partitions

You can use fdisk or parted to recreate partition table on the target machine. I used

fdisk /dev/vda

I created the /boot partition on /dev/vda1 with the exact same size as on the source machine (fdisk -l /dev/vda on source). Then I created /dev/vda2, which was a little bit larger.

I recreated the volumes as they were – volume group name was vg0, I had two logical volumes: root and swap. Use vgdisplay and lvdisplay to understand your exact situation

Then on target:

# create physical volume, add it into a newly created volume group
pvcreate /dev/vda2 
vgcreate vg0 /dev/vda2
# create and initialize swap
lvcreate -L 4G -n swap vg0
mkswap /dev/vg0/swap
# create and initialize root volume, make sure the size is at least the source size
lvcreate -L 50G -n root vg0

Copy the /boot partition

Setup networking on target (System Rescue CD), make note of your IP address.

Then prepare for receiving on the target machine. Replace TYPEASTRONGENCRYPTIONPASSWORDHERE with some random password you will use for the transfer:

nc -l -p 444 | openssl aes-256-cbc -d -salt -pass pass:TYPEASTRONGENCRYPTIONPASSWORDHERE -md sha256 | pigz -d | dd bs=16M of=/dev/vda1

Note that we don’t use SSH, it is slow with buffers, it would take ages, especially the large volumes.

On source machine, first remount /boot read only, transfer and then remount readwrite. Change the IP address of the target:

mount -o remount,ro /boot
pv -r -t -b -p -e /dev/vda1 | pigz -1 | openssl aes-256-cbc -salt -pass pass:TYPEASTRONGENCRYPTIONPASSWORDHERE -md sha256 | ncat TARGETIP 444
mount -o remount,rw /boot

The pv command will give you a fancy progress bar with ETA, neat!

After receiving on the target, just check the filesystem

fsck.ext4 /dev/vda1

LVM snapshots

Now the funny part, creating the snapshot and copying.

This plan can work only if you have some free space in volume group. In one case I used an ugly hack, but that could backfire. Snapshots are thin and only require the amount of space that would cover all the writes between the time the snapshot was taken and now. I did not have space on one machine, so after transferring /boot partition to the target, I deleted few old kernels from it and used it as a volume (losetup, pvcreate, vgextend – I am not writing exact commands, you really should understand what you are doing here). Warning – only do this on partitions that are not part of LVM!

Assuming we now have space in volume group vg0 on source machine, let’s create the snapshot:

lvcreate --snapshot --name transfer_snap --size 1G /dev/vg0/root

This snapshot does not have to be consistent, you can do it while the machine is doing everything. Check from time to time if the snapshot is not running out of space using lvs command.

Prepare for receiving on the target (we only changed the destination block device to /dev/vg0/root, no snapshots on the target are required):

nc -l -p 444 | openssl aes-256-cbc -d -salt -pass pass:TYPEASTRONGENCRYPTIONPASSWORDHERE -md sha256 | pigz -d | dd bs=16M of=/dev/vg0/root

Now send over the snapshot from source, changing the TARGETIP:

pv -r -t -b -p -e /dev/vg0/transfer_snap | pigz -1 | openssl aes-256-cbc -salt -pass pass:TYPEASTRONGENCRYPTIONPASSWORDHERE -md sha256 | ncat TARGETIP 444

This can take a long time, go and enjoy some movie. If there are more volumes, repeat.

Now make sure you have lvmsync installed and in path on both source and target. Also make sure you can ssh from source to target as root (in System Rescue CD it only involved setting up the password using passwd).

If you tested everything, now is the time to do some shutdowns. Use systemctl to shutdown a webserver, database server, if you use Bitcoin lightning, make sure to shutdown lnd or anything else that could make the system inconsistent. This will hopefully be brief, but let’s hope it’s 4am and no one uses the server anyway:).

Then run the sync on source, replace target IP with the target’s IP:

lvmsync /dev/vg0/transfer_snap TARGETIP:/dev/vg0/root

It will ask for root’s password on the target and sync the changes.

Bootloader and networking

Now we will check the filesystem and setup bootloader on target:

fsck /dev/vg0/root # you might invoke a particular fsck for your filesystem type
mount /dev/vg0/root /mnt
chroot /mnt # after this, we are in the context of the target machine
mount -t devtmpfs dev /dev
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mount /dev/vda1 /boot # make sure grub sees the correct boot partition
/usr/sbin/grub2-install --boot-directory=/boot /dev/vda
export PATH=$PATH:/usr/sbin
/usr/sbin/grub2-mkconfig -o "$(readlink -e /etc/grub2.cfg)" # especially this part is Centos/RHEL specific. On ubuntu, you would do something like update-grub

Now, the last part is checking the network settings, change /etc/sysconfig/network-scripts for your ethernet adapter and make sure you have the correct network settings.

Then still on target:

umount /dev
umount /proc
umount /sys
umount /boot
exit # from chroot
umount /mnt
reboot

Forward ports

Hopefully, you have a nice new machine that booted and everything works (if it booted, there’s a good chance that it does, unless you forgot old IPs somewhere).

We need to change any DNS entries to the new machine. I prefer testing out the new installation without changing DNS. On the old source machine:

yum install socat
# let's forward 80 and 443, leave this running in screen
socat -d -d -lmlocal2  TCP4-LISTEN:80,reuseaddr,fork,su=nobody TCP4:TARGETIP:80 &
socat -d -d -lmlocal2  TCP4-LISTEN:443,reuseaddr,fork,su=nobody TCP4:TARGETIP:443 &

Now test the website or any other services. If you are happy with it, change the DNS, keep the socat running, so the web works for those who have cached the old IP address.

The last thing to do on the old machine to cleanup is to make sure the web/database/lnd/… don’t start again (use systemctl disable) and delete the snapshot, otherwise the volume group would probably run out of space soon:

lvremove /dev/vg0/transfer_snap

The end

I hope you enjoyed this tutorial. What I especially like about this approach is that it is completely from “within” the machine, except for booting System Rescue CD on target which most hostings allow you to do.

I liked learning about lvmsync, pv (the piped-progress bar that I will use for everything from now on), pigz (parallel fast piped gzip), openssl passthrough encryption (which is way faster than SSH, because of buffers) and relearning socat again.

The downtime was very short, the services were down for the time between starting lvmsync and rebooting (after installing the bootloader). In one case I had to reinstall the kernel package.

Hope you had fun, if you found this tutorial useful, please head to Support me page and send me some crypto-love!