HowTo: VM Copy with Snapshots and Similar Tasks
Posted: 10. Oct 2010, 22:28
Introduction
I've spent a good amount of time searching on the forums and elsewhere for a method of copying virtual machines between different VirtualBox installations while preserving snapshots. All that I've ever seen as a solution is to merge all the snapshots together into a single VM and then export it. But this is completely unnecessary. All that you need is enough boldness to dive into the XML file and edit it manually, as we will see. Using the methods outlined herein, I have achieved the following things:
Let me mention that we are playing with fire a bit here. Having multiple VM's referencing the same disks is not designed behavior, which is why you can't do it with built-in tools. We have to be very careful about deleting snapshots etc when they are referenced by multiple VM's. VirtualBox will not help you once you go off the reservation in this manner. So make sure you give it a good amount of thought before you go deleting snapshots after Cloning VM's (#2 above).
The XML Files
Each VM is completely defined in its own XML file. All of the snapshots are defined in order in the file as well as preferences for each snapshot, the current snapshot, system setup, extra data, etc. Since the file is XML, it is remarkably easy to follow when viewed in a text editor. I personally recommend Notepad++ since it allows you to fold up irrelevant sections of the code to get a better understanding of what's going on.
The Snapshots are defined in a tree structure in the XML just as they are viewed in the VirtualBox GUI. Each snapshot has a GUID to identify it as well as a differencing disk with a separate GUID that is associated with it. Contained within each snapshot tag is another <Snapshots> tag that defines the children of the snapshot. The nested XML snapshot definition and the referenced differencing vdi files are the two essential things that make up the snapshot scheme.
In addition to the VM XML file that defines the VM, VirtualBox itself has its own XML file that is probably in the .Virtualbox directory of your home directory. For Windows users, this is "Documents and Settings\Username\.Virtualbox" in all likelihood. This XML file is the other side of the VM definition. While the VM XML fully encapsulates the VM itself with all of its snapshot information and hard drive referencing information, the VM must still be declared to exist in the main VirtualBox XML. Each declaration of a VM points to the appropriate VM XML file so that VirtualBox may retrieve the appropriate settings for the VM. Also, the main VirtualBox XML contains the definitions of all hard drives. In the VM XML files, the hard drives are referenced by UUID, NOT by path. So we have to make an association between UUID's and actual paths to actual files in the VirtualBox XML.
So for our purposes, here are the two files of importance and their contents.
GUID's can prove to be the bane of our existence when we're doing things like this with the XML. If you're trying to simply copy a VM made on one machine to a VirtualBox install on another machine, then the odds are extremely low that you will have any problems with GUID's. That's the whole point of them actually. "The primary purpose of the GUID is to have a totally unique number. Ideally, a GUID will never be generated twice by any computer or group of computers in existence." [Wikipedia] That's pretty cool to think about in general, just like RSA cryptography.
That being said, if we are trying to make clones of VM's and other such tasks, we will be trying to copy an existing VM into the same VirtualBox Main XML file. Therefore, the GUID's will not have been generated at different times, and you can run into a lot of problems. Virtual hard drives have their GUID's hard-coded into the file (as you might already know). And the only way to change that is to use the clonedvi program provided with VirtualBox, which doesn't work on differencing vdi's. However, if you are trying to copy a hard drive with the same GUID, odds are that it also has the same contents. If this is the case, then you can simply use the previously defined identical hard drive with its original GUID. If somehow you end up with hard drives with identical GUID's and different data then you are just out of luck. You probably did something stupid before thinking about it to get to that point. Most of the time, hard drives will not be your problem. For all other GUID's (VM's, Snapshots, etc), the GUID's are stored simply in the XML files and can be modified there without any problems. Just make sure you make the appropriate changes everywhere the GUID occurs in all relevant XML files. Everything you will want to do is probably still possible. It may just be tedious to accomplish and require some trial and error.
Task 1: Complete VM Copy from another Machine
Let's start with what is probably the simplest task outlined above. To copy a VM from one machine to another we will need all of the components of that VM, namely:
Task 1.1 The VM XML
You hardly need to do anything to the VM XML. Everything should be left as is, because the hard drives are referenced by GUID. We will take care of that in a second. The only change that might need to be made here is if any of your snapshots have media that you will not be copying over (i.e. iso files). Just go into the <DVDDrive passthrough="false"> <HostDrive src="/dev/sr0"/> area and get rid of all references to iso files. Just delete the child tag the references the iso file. In fact, run a search for iso and delete all references to it by deleting the whole child tag. This may need to be done for all snapshots. Of course, if you want to keep your iso files, then just add them to the target VirtualBox Main XML like the hard drives (Task 1.2.2) and leave this file alone.
Task 1.2.1 The VM Definition
This is the easier portion of the XML. Go into the <MachineRegistry> section of the VirtualBox Main XML. There you will see all of your VM's laid out in all their glory. Choose the appropriate line using the name of the VM as a guide. For example you might want to choose a line like this one.
This will need to be copied into your new VirtualBox Main XML file. Put it in the same <MachineRegistry> section and make sure that there is no duplicate GUID. There shouldn't be. If there is, then just make some change to it and make the same change everywhere in your VM XML file that you were going to copy as is. Put your VM XML file in the directory specified here. If that directory doesn't exist, copy it or change it. Just make sure that the VM XML file makes it in there.
Task 1.2.2 The Hard Disk Definition
Since the previous part was the easy part, this must be the harder one. Usually it's not that big of a deal when copying a full VM. You just need to figure out which hard drive definition to copy. You need to know your base hard drive, which you can find by looking at the GUI before hand or looking through the VM XML file. Or maybe you just named it intelligently and can tell what it is. The snapshot hard drives are usually saved in the "Snapshots" folder of the VM in question. So your base vdi might be in one directory, and your snapshots might be in a different directory. That doesn't matter as long as everything is defined properly in this section. In this example, I will copy the following lines
Try to read around the wordwrap. As you can see, this hard drive definition starts with the aptly named "Windows Box.vdi" and then drills down into snapshot hard drives (differencing hard drives) that are stored in the VM's Snapshots directory. These are not named in such a user-friendly manner. Their names correspond to their GUID's, as you may have noticed. This is a pretty good idea, because it keeps the automatically named hard drives from ever having a duplicate name in the same directory (or anywhere for that matter). And since everything is corralled here in this XML, it really doesn't matter what the names are. What I've shown here is a very simple example with just one snapshot, but yours may have many different snapshots in various different tree structures. Not to worry. Just make sure that you copy everything from the start of the definition (<HardDisk uuid=...>) to the corresponding end tag (</HardDisk>). Keep in mind that you need to have an end tag for every start tag unless the tag is like the snapshot above. In that case, the start tag is its own end tag, because it has no children. This is denoted by having a "/>" at the end of the tag. Tags with children have only a ">" to close the beginning tag. Of course, if you're using Notepad++, it is easy to see which closing tags go to which opening tags due to the nice dotted lines that it draws for you.
The easiest thing to do is to copy all of the relevant vdi's into the exact same location on the target machine, but this may not be possible. If you want to move any of the vdi's, you just have to change the "location=..." portion of the tags that point to them. This is the only change that needs to be made, but if you don't make it VirtualBox will complain and refuse to boot your VM. Make sure you don't mess up the parent child relationship of anything. Otherwise you will have a terrible time trying to get it right again. Copy and Paste are your friends.
You can probably figure out that the copied lines should go in the <MediaRegistry> <HardDisks> section of the target VirtualBox Main XML.
Task 1.2.3 Hard Drives
Hopefully your hard drives are all pretty much in the same place. The snapshots are usually all in the VM's Snapshots directory. The base vdi may be somehwere else, but that's not too bad if it's just one. There are no tricks here. You can't do much with the vdi files. Simply copy them into the appropriate locations as defined in the previous step. Don't use any VirtualBox tools. Just a simple Windows copy is all we need here. Make sure you get them all. Check each one with the definition in the previous step if you have problems.
Task 1.2.4 Start VirtualBox and Declare Victory
If you've copied everything into the right directories and updated your XML file with VBoxSVC killed, then you should be ready to go. Start up VirtualBox and look for your new shiny VM. It should be there. Look at the snapshots. They should all be present and accounted for. If you would like, take a look at the Virtual Media Manager and verify that your hard drive is intact with no missing pieces. Try restoring some snapshots and booting the VM. If there are any problems, the VirtualBox error messages are usually very helpful. Watch out for stray iso file references etc that you didn't copy over or delete. The process should be relatively painless. If everything goes to hell, then kill VBoxSVC, copy your backups back into place, and start over. I am doing this guide from memory, so it's possible that I left something out. Not likely though.
This is all I am going to write for now. The more complicated tasks mentioned above might be clear to you based on what we've done. It's really all the same, except you don't copy as much, and sometimes you have to change GUID's everywhere. One thing off the top of my head about cloning VM's to run at the same time. Before starting restore a snapshot, and after you complete the clone restore a snapshot again in both VM's. You have to be careful that the two VM's don't reference the same writable hard drive. What would be best would be if someone would write a program to do all of this XML management for you, especially since cloning VM's can get messy when you start adding snapshots that won't show up in both VM's. It would be nice to keep everything synchronized automatically. It can be done by hand if need be. Just copy and paste. Hope this information is helpful.
Bill
I've spent a good amount of time searching on the forums and elsewhere for a method of copying virtual machines between different VirtualBox installations while preserving snapshots. All that I've ever seen as a solution is to merge all the snapshots together into a single VM and then export it. But this is completely unnecessary. All that you need is enough boldness to dive into the XML file and edit it manually, as we will see. Using the methods outlined herein, I have achieved the following things:
- 1. Copy an entire VM with snapshot tree from one computer to another
2. "Check out" a single branch of a snapshot tree and copy it to another computer
3. "Cloned" a VM so that two snapshots from the same VM can be run at the same time (as separate VM's)
4. Copied snapshots into a VM from another VM (which happened to be a clone)
Let me mention that we are playing with fire a bit here. Having multiple VM's referencing the same disks is not designed behavior, which is why you can't do it with built-in tools. We have to be very careful about deleting snapshots etc when they are referenced by multiple VM's. VirtualBox will not help you once you go off the reservation in this manner. So make sure you give it a good amount of thought before you go deleting snapshots after Cloning VM's (#2 above).
The XML Files
Each VM is completely defined in its own XML file. All of the snapshots are defined in order in the file as well as preferences for each snapshot, the current snapshot, system setup, extra data, etc. Since the file is XML, it is remarkably easy to follow when viewed in a text editor. I personally recommend Notepad++ since it allows you to fold up irrelevant sections of the code to get a better understanding of what's going on.
The Snapshots are defined in a tree structure in the XML just as they are viewed in the VirtualBox GUI. Each snapshot has a GUID to identify it as well as a differencing disk with a separate GUID that is associated with it. Contained within each snapshot tag is another <Snapshots> tag that defines the children of the snapshot. The nested XML snapshot definition and the referenced differencing vdi files are the two essential things that make up the snapshot scheme.
In addition to the VM XML file that defines the VM, VirtualBox itself has its own XML file that is probably in the .Virtualbox directory of your home directory. For Windows users, this is "Documents and Settings\Username\.Virtualbox" in all likelihood. This XML file is the other side of the VM definition. While the VM XML fully encapsulates the VM itself with all of its snapshot information and hard drive referencing information, the VM must still be declared to exist in the main VirtualBox XML. Each declaration of a VM points to the appropriate VM XML file so that VirtualBox may retrieve the appropriate settings for the VM. Also, the main VirtualBox XML contains the definitions of all hard drives. In the VM XML files, the hard drives are referenced by UUID, NOT by path. So we have to make an association between UUID's and actual paths to actual files in the VirtualBox XML.
So for our purposes, here are the two files of importance and their contents.
- VirtualBox Main XML File
- 1. Declarations of VM's (1 line references to the VM XML file)
2. Definitions of Virtual Hard Drives
- 1. All Settings for the VM
2. Snapshot Tree Definition
- 1. Declarations of VM's (1 line references to the VM XML file)
- 1. Make a backup of all XML files that will be edited.
2. If you want to be really safe, make backups of the VDI files as well (including Snapshots directories for each VM)
3. Close all VirtualBox GUI's and VM's
4. Kill the VBoxSVC service. This is very important, because this service keeps copies of the XML files in memory and will OVERWRITE what you have just spent a lot of energy doing if you don't watch out.
GUID's can prove to be the bane of our existence when we're doing things like this with the XML. If you're trying to simply copy a VM made on one machine to a VirtualBox install on another machine, then the odds are extremely low that you will have any problems with GUID's. That's the whole point of them actually. "The primary purpose of the GUID is to have a totally unique number. Ideally, a GUID will never be generated twice by any computer or group of computers in existence." [Wikipedia] That's pretty cool to think about in general, just like RSA cryptography.
That being said, if we are trying to make clones of VM's and other such tasks, we will be trying to copy an existing VM into the same VirtualBox Main XML file. Therefore, the GUID's will not have been generated at different times, and you can run into a lot of problems. Virtual hard drives have their GUID's hard-coded into the file (as you might already know). And the only way to change that is to use the clonedvi program provided with VirtualBox, which doesn't work on differencing vdi's. However, if you are trying to copy a hard drive with the same GUID, odds are that it also has the same contents. If this is the case, then you can simply use the previously defined identical hard drive with its original GUID. If somehow you end up with hard drives with identical GUID's and different data then you are just out of luck. You probably did something stupid before thinking about it to get to that point. Most of the time, hard drives will not be your problem. For all other GUID's (VM's, Snapshots, etc), the GUID's are stored simply in the XML files and can be modified there without any problems. Just make sure you make the appropriate changes everywhere the GUID occurs in all relevant XML files. Everything you will want to do is probably still possible. It may just be tedious to accomplish and require some trial and error.
Task 1: Complete VM Copy from another Machine
Let's start with what is probably the simplest task outlined above. To copy a VM from one machine to another we will need all of the components of that VM, namely:
- 1. The VM XML File
2. The relevant portions of the VirtualBox Main XML File
3. The relevant VDI files that contain the actual data for the VM
Task 1.1 The VM XML
You hardly need to do anything to the VM XML. Everything should be left as is, because the hard drives are referenced by GUID. We will take care of that in a second. The only change that might need to be made here is if any of your snapshots have media that you will not be copying over (i.e. iso files). Just go into the <DVDDrive passthrough="false"> <HostDrive src="/dev/sr0"/> area and get rid of all references to iso files. Just delete the child tag the references the iso file. In fact, run a search for iso and delete all references to it by deleting the whole child tag. This may need to be done for all snapshots. Of course, if you want to keep your iso files, then just add them to the target VirtualBox Main XML like the hard drives (Task 1.2.2) and leave this file alone.
Task 1.2.1 The VM Definition
This is the easier portion of the XML. Go into the <MachineRegistry> section of the VirtualBox Main XML. There you will see all of your VM's laid out in all their glory. Choose the appropriate line using the name of the VM as a guide. For example you might want to choose a line like this one.
Code: Select all
<MachineEntry uuid="{3287d66b-7c64-4c3a-b513-1d21c4bf0e3b}" src="Machines/Windows Box/Windows Box.xml"/>Task 1.2.2 The Hard Disk Definition
Since the previous part was the easy part, this must be the harder one. Usually it's not that big of a deal when copying a full VM. You just need to figure out which hard drive definition to copy. You need to know your base hard drive, which you can find by looking at the GUI before hand or looking through the VM XML file. Or maybe you just named it intelligently and can tell what it is. The snapshot hard drives are usually saved in the "Snapshots" folder of the VM in question. So your base vdi might be in one directory, and your snapshots might be in a different directory. That doesn't matter as long as everything is defined properly in this section. In this example, I will copy the following lines
Code: Select all
<HardDisk uuid="{88aca96d-dd9f-45b6-a8a1-69c6905d9cf8}" location="HardDisks/Windows Box.vdi" format="VDI" type="Normal">
<HardDisk uuid="{0ae91875-203c-45fe-965c-3ca619d7f308}" location="/home/bill/.VirtualBox/Machines/Windows Box/Snapshots/{0ae91875-203c-45fe-965c-3ca619d7f308}.vdi" format="VDI"/>
</HardDisk>The easiest thing to do is to copy all of the relevant vdi's into the exact same location on the target machine, but this may not be possible. If you want to move any of the vdi's, you just have to change the "location=..." portion of the tags that point to them. This is the only change that needs to be made, but if you don't make it VirtualBox will complain and refuse to boot your VM. Make sure you don't mess up the parent child relationship of anything. Otherwise you will have a terrible time trying to get it right again. Copy and Paste are your friends.
You can probably figure out that the copied lines should go in the <MediaRegistry> <HardDisks> section of the target VirtualBox Main XML.
Task 1.2.3 Hard Drives
Hopefully your hard drives are all pretty much in the same place. The snapshots are usually all in the VM's Snapshots directory. The base vdi may be somehwere else, but that's not too bad if it's just one. There are no tricks here. You can't do much with the vdi files. Simply copy them into the appropriate locations as defined in the previous step. Don't use any VirtualBox tools. Just a simple Windows copy is all we need here. Make sure you get them all. Check each one with the definition in the previous step if you have problems.
Task 1.2.4 Start VirtualBox and Declare Victory
If you've copied everything into the right directories and updated your XML file with VBoxSVC killed, then you should be ready to go. Start up VirtualBox and look for your new shiny VM. It should be there. Look at the snapshots. They should all be present and accounted for. If you would like, take a look at the Virtual Media Manager and verify that your hard drive is intact with no missing pieces. Try restoring some snapshots and booting the VM. If there are any problems, the VirtualBox error messages are usually very helpful. Watch out for stray iso file references etc that you didn't copy over or delete. The process should be relatively painless. If everything goes to hell, then kill VBoxSVC, copy your backups back into place, and start over. I am doing this guide from memory, so it's possible that I left something out. Not likely though.
This is all I am going to write for now. The more complicated tasks mentioned above might be clear to you based on what we've done. It's really all the same, except you don't copy as much, and sometimes you have to change GUID's everywhere. One thing off the top of my head about cloning VM's to run at the same time. Before starting restore a snapshot, and after you complete the clone restore a snapshot again in both VM's. You have to be careful that the two VM's don't reference the same writable hard drive. What would be best would be if someone would write a program to do all of this XML management for you, especially since cloning VM's can get messy when you start adding snapshots that won't show up in both VM's. It would be nice to keep everything synchronized automatically. It can be done by hand if need be. Just copy and paste. Hope this information is helpful.
Bill