CloneVDI tool - Discussion & Support

[mpack]

I'm glad CloneVDI worked for you.

The repair calculations are relatively straightforward but I wouldn't recommend that anyone try it manually without experience (and there should be no need, now that CloneVDI automates it).

From the current VDI size you can calculate the number of 1MB blocks already allocated. Normally the header is 2MB (if the capacity is less than 256GB that is), so N-2 is the number to patch the nAllocatedBlocks header field with. Older VDIs had different rules, header blocks were multiples of 4K but image blocks were still 1MB.

Scanning the block map to find the end - you'll see where it transitions from 0x?????????? to solid 0x00000000. The address of the transition gives you the capacity of the drive. E.g. if we treat the block map as an array of DWORDs and M is the index of the first 0x00000000 after the end of the blockmap then the drive has a capacity of M MB. So get the user to create another VDI with the same capacity to serve as a repair patch.

That's usually all you need to know to reconstruct a usable header for a stand-alone VDI.

Note that these are hueristics, i.e. they usually work, but there is no guarantee. If the block map is trashed then nothing can be done. If the VDI is a snapshot then the prognosis is poor, because the block map patterns are different plus the UUID links are critical and can't be guessed without additional technical info which is often not available from someone who decided to use snapshots.

[RC1965]

Thank you for making this tool mpack. Here is a summary of my experience.

I am using Virtualbox 5.2.8r121009. I had a Windows 10 PC running as a guest with a snapshot. After several days of normal operation today I received the dreaded error message (sorry I didn't write down exact wording but it was something like VD_HEADER_...) and it wouldn't start my guest OS. [For the record I want to mention that Virtualbox has NEVER before corrupted any of my VMs before this and I've been using it for 10+ years].

Luckily I found your post not long after beginning my search for a solution. In my VM, I had a primary 60GB VMDK and a secondary 1TB VMDK but the snapshot was only on the primary. I kept my VM, primary disk and snapshot disks all in separate custom locations (as opposed to default folders). After making a full backup of my primary VMDK, I ran CloneVDI and it failed with the message, "differencing disks are not supported". Here again I got lucky in that I had read your release notes.txt very carefully. So I moved the base VMDK in the same folder as the snapshot VMDK and tried again. CloneVDI started creating a new VDI for me in the destination folder.

I did not change any options in CloneVDI before starting it except for specifying the snapshot VMDK in the source and a destination folder which was on a different drive. The destination VMDK had a size of 39GB which alarmed me a bit but when I found out later that CloneVDI actually kept my dynamically expanding drive intact and the virtual size was in fact 60GB. Amazing.

Afterwards, I created a new VM in Virtualbox and added the new VDI, adjusted other settings for this VM and started it up and it started up without any issues. I have marked the previous one for deletion in two weeks, if I found no problems with this one, as you had suggested elsewhere.

Thanks again for saving the day.

[mpack]

It sounds like you read the docs and did everything the best way, so well done. My only surprise is that you didn't continue to have trouble with the header damage, since the CloneVDI header repair feature doesn't know anything about VMDK. I guess it must have been a non-fatal header problem such as an inconsistent backup grain table. You can get that with VMDK after a crash or power cut.

I suggest that in future you stick to VDI. The structure is competent but less flexible than VMDK, and hence far easier to repair. Plus it's the native format in VirtualBox and so supports all VirtualBox modes.

[RC1965]

You are right again. The message was about inconsistent grain. I should have written all the messages down but I just kept moving on with the fix.

The only reason I choose VMDK is so I don't have to convert if I ever have to switch to VMware Workstation. But I understand it's not so hard to do so. I will start using native formats. Thanks.

I know this has nothing to do with CloneVDI (which saved my VM) but my Windows 10 VM lost it's activation in the whole process and it simply would not activate with that same key I used before (from Windows 8.1). I had to purchase a new low cost key from eBay to activate.

[mpack]

The Windows 10 activation problem must have been a separate issue, such as a change of CPU, because AFAIK Win10 doesn't treat disk UUID (the only feature CloneVDI has control over) as a significant factor in the hardware signature any more: the Win10 developers are well aware that disk imaging and disk replacement happens, so the signature now concentrates on "motherboard features".

[kwmaeng91]

Hi, thanks for the great software.
Unfortunately, this did not work for me as I expected.
I am running a 64-GB ext3 archlinux in my VM, which actually contains only 8GB of files.
I tried compacting, but did not help much. (result was from 40GB to 39GB or so)
Is it because I am using arch? Or is the result normal? (I was expecting something around 8GB to be the compacted result) Or maybe is it because I have snapshots?
Searched for a while but could not find an answer.

Is there any suggestion so that I can look into?
I also tried zeroing and using VBoxManage modifyhd --compact, which also did not shrink the result.

[mpack]

I tried compacting, but did not help much. (result was from 40GB to 39GB or so)
Is it because I am using arch?

I have no idea if this was because you were using Arch, since I know nothing about Arch Linux, other than that it exists. CloneVDI compaction will work in partitioned spaces of the disk as well as inside partitions which use a recognized filesystem. Of course it must use a recognized partition scheme too (i.e. MBR is the only such recognized by current releases of CloneVDI).

So, which partition scheme does you guest use? how many partitions? and what filesystem in each partition? If you know the answers to those questions then you should be able to predict the effectiveness of CloneVDI compaction.

Compaction by zeroing (I notice that you didn't say how you did this) works for CloneVDI too, but again is only effective if you zero out the correct partition(s).

I suspect that your estimate of the drive containing only 8GB data is incorrect, and will be based on looking at one small partition among several. Ignoring for example a swap partition which many Linux's have, and perhaps other special purpose partitions which Arch creates (like I said, I have no knowledge of what Arch does).

[kwmaeng91]

I just deleted all the snapshots and tried again.
This time it gave me 40G -> 10G!!
Got a possible hint from here that snapshots might be a problem: https://wiki.archlinux.org/index.php/Vi ... tual_disks

Note: If your virtual machine has snapshots, you need to apply the above command on each .vdi files you have.

I wasn't sure what it was saying about "apply on each .vdi", but anyway it worked!

I wonder if this is abnormal, or CloneVDI does not work if there are snapshots taken before cleaning up the disk space.
(I don't know much about how snapshots work, but maybe to you it is obvious)

Anyway, thank you for the wonderful work!
Worked like charm!

[mpack]

Got a possible hint from here that snapshots might be a problem:

Um, I'm afraid that should not have been new information. Search for snapshots in the CloneVDI release notes. Or read the resizing FAQ on this site.

Basically, if you wanted CloneVDI to take account of the snapshot rather than the base VDI then you should have selected the snapshot VDI as the source, and not the base VDI. You can select any point in time by choosing the appropriate snapshot (the base VDI represents the oldest possible state), but normally you would choose the newest one.

[Timo64]

Hi Mpack,
thanks a lot for your great and thorough work!

1. I installed your tool in order to compact my +60GB snapshots, and I carefully read the release notes. According to those notes, choosing the option to *keep* the UUID of my old VDI file and selecting the same file name both for source and destination, I expected to end up with a brand new cloned VDI having the name of the old one, while the old one should have been renamed to "Original of ...". But actually the opposite happened: the cloned VDI took the name of "Clone of..." while the old one kept its old (original) name without any prefix. That's no big deal and I renamed them by myself, but I wonder how the release notes could be so blatantly inaccurate..?
2. However, I started the VM using the newly created clone (having the name and UUID of the old VDI, I didn't change anything in the VM configuration) and the VM works fine, just like before. BUT: I expected the snapshots of my old VDI to "disappear", i.e. to be no longer in use and not even to be displayed by Virtualbox. Instead, the VM configuration still relies upon them. The Configuration panel shows all five snapshots in a chain, and my current position is still below "Snapshot 5"...
3. I found a hint in this forum that I should have used NOT the basic VDI as source, but rather the last snapshot. So I CloneVDI-cloned that snapshot and changed its garbled name to the name of the orignal (basic) VDI disk, then moved it to the parent folder (i.e. the level above the "Snapshots" folder). But still this clone relies upon all the older snapshots! I tried to change the name extension of its snapshot files to ".vdi-old", hence making them invisible to Virtualbox. But now my VM didn't work at all: I changed back the file extensions from ".vdi-old" to ".vdi", and now the VM is up and running again - hence, the snapshots are still needed for the VM to work.

So: what was the point of using CloneVDI..? I probably missed something and it's all my fault, but I thought I had followed your instructions to the letter - nevertheless I ended up with a useless clone that still depends on all five snapshots. Could you please enlighten me what I should do differently?
Thanks a lot in advance, cheers,
Timo
P.S. I don't think it really matters, however my host is Win 10 1803, and the guest is Win XP SP3

[socratis]

You have a recipe that uses a VDI with snapshots 1, 2, 3. If you change the VDI, the recipe doesn't change; you have a VM that uses a VDI (new one) with snapshots 1, 2, 3. You din't really expect that by changing the VDI in use that the recipe of the VM would change, right?

What you missed, is this part from the instructions:

Q. I've used CloneVDI to clone my source disk.  How do I run the cloned VM under VirtualBox?
A. In fact you don't have a cloned VM, all you have is a cloned hard disk which you are now free
   to either mount in an existing VM, or create a new VM around. A more precise answer to this
   question depends on how and why you made the clone.
   o If you cloned a source VDI to a dest VDI with the same name, same UUID, and in the same folder
     then you don't need to do anything else - from now on VirtualBox uses the new VDI in all
     VMs which used the original VDI.
   o If you want to create a new VM around the cloned VDI then do the following :-

You thought that the part in blue would pretty much make the Snapshots vanish magically. Well, that's not going to happen, see my opening statement.

What you need to do is the second part; create a new VM around the cloned VDI.

[mpack]

That's no big deal and I renamed them by myself, but I wonder how the release notes could be so blatantly inaccurate..?

The feature is seldom used, perhaps it has been broken. Though more likely you supplied slightly different filenames or different paths for source and destination. Unless you supply specifics it is impossible to say.

[Timo64]

Thank you, Socratis, for this clarification. So it turns out the only way I can get rid of the snapshots and reduce wasted disk space is by simply cloning my VM with the help of the Virtualbox cloning wizard. Actually, it's not even necessary to "create a new VM around the cloned VDI", it's a lot faster and easier to just fully clone the old VM based on its last state. That's what I could and should have done in the first place. I then deleted the original VM, freeing up 50+ GB of disk space, and the lean new clone works like a charm. I'm grateful for mpack's efforts but I have the impression the only real use of the CloneVDI Tool is to squeeze the VM disk a little and to make it possibly a bit faster and safer by defragmentation.
Timo64

[socratis]

but I have the impression the only real use of the CloneVDI Tool is to squeeze the VM disk a little and to make it possibly a bit faster and safer by defragmentation.

I'm sorry, but you have the wrong impression. CloneVDI is much more than that. First of all, when CloneVDI came out there was no clone VM in VirtualBox. Second, the options that CloneVDI offers are still not part of VirtualBox. So, if cloning a VM worked for your scenario, that's cool. But, please do not dismiss so easily the functionality of CloneVDI, just because it didn't fit in your usage scenario. Just count the "Thank you" messages in this thread alone...

In fact I wish sometimes that the functionality of CloneVDI was adopted at some point by the main VirtualBox program, but it's not me that makes those decisions, at either end.

[Manea]

Hi, i have a small problem that i think CloneVDI tool could help with but I'm not sure.

I have a server running Virtualbox and several VM's in it. One of the disk on the system is unstable so we need to install a new disk and while at it we want to upgrade Linux version on the base server that runs Virtualbox.

Most of the VM's are relatively small and were fast to just do a full clone using Virtualbox's own cloning tool while the servers were offline, but for one of our servers that isn't an option.

The size of one of our VM is over 400GB, and consists of 2 base disks, (disk1.vmdk & disk2.vmdk) and 7 *.vmdk snapshots.
This server is crucial to our services so we cannot take it offline for the duration of the cloning (which would be several hours).

So the question is, can i use CloneVDI tool to create a single .vdi image of the server i previously described and create a new VM and use this new cloned .vdi as an "existing disk" to start it up?
Thanks.