CloneVDI tool - Discussion & Support

[mpack]

I don't recognize the issue or the error message. I would look to the media you are using to receive the clone.

[HuaMin]

What would you like to further check for such error? The original Vbox is working fine.

[mpack]

I'm not interested in discussing the error here, unless you can demonstrate a problem caused by CloneVDI.

[HuaMin]

Did you or anyone ever clone such big Vbox using CloneVDI? Did you try it fine or not for such big size? My original Vbox is 680 GB while the newly cloned one is 559 GB, which is not bootable right now.

[mpack]

I personally have never cloned such a large file, but the size is not near any suspicious thresholds. I know CloneVDI can handle images >4GB (an issue if any intermediate arithmetic was done in 32bits). 2TB would be the next interesting size due to LBA addressing limits. 8GB is the limit for legacy CHS addressing. 680GB is nowhere near any of those. However, as I say in the release notes, a bug is always possible, especially if you do something unusual. In this case I don't have an 680GB vdi to test, creating one would be quite a lot of work, and no-one else has reported that CloneVDI's most basic function - copying data from A to B - ever fails, so right now I'm not inclined to invest time to look at this problem.

I suspect "pilot error". Though the size doesn't make me think of a CloneVDI bug, it does make me think that you need to find a lot of free disk space for the clone, and even more space again if you copied the file again, outside of CloneVDI. Another thought is that if you sent the clone to a FAT32 formatted external drive then you ought to have got write errors during cloning.

Anyway such problems can never be catastrophic: you always have the original to return to.

[WalterC]

Hi Don, I need a little help with a clonevdi error.
I lost a snapshot file while trying to delete the tree and get rid of the snapshots. The VB merging process decided to use my C drive instead of the W: drive where the vhd files and snapshots are stored. I didn't know this until if failed because of lack of disk space. That failure apparently killed the second snapshot file in the link.

Structure in error

vboxstructure.JPG (39.12 KiB) Viewed 6522 times

After hours of trying different things with no success, I finally decided to try and skip over the missing snapshot by pointing the missing snapshot's child to it's parent. I have nothing to lose at this point. I used clonevdi to doublecheck the UUIDs before I started. It had no problem reading the VHD.

happy.JPG (34.83 KiB) Viewed 6522 times

I then changed the parent UUID by copy past from the actual parent UUID to make sure I got all those hex codes correct.

After I saved the changes, I wanted to double check with clonevdi and I got this disappointing error.

nothappy2.JPG (65.72 KiB) Viewed 6522 times

Is there something else I need to do besides changing and saving the 16 bytes for parent UUID. Why does clonevdi think the file is invalid now? I read some of your other posts on this and I thought it was clear that's all I needed to do.

Thanks for your help and for clonevdi.

-Walter

[mpack]

I don't recall ever giving advice on patching the UUID fields in a VHD header! With VDI it would be fairly easy, but VHD has two copies of the header (a header and an optional footer - the latter ignored by CloneVDI if the header is present), plus each header has a 32bit checksum field to protect its contents from modification. It is the latter failure which will certainly cause CloneVDI to reject your modified VHD.

You should download the VHD format specification if you need to understand it. However, after a quick check I can quickly tell you that the checksum (32bits) is the 1s complement of the sum of the bytes in the dynamic header (not counting the checksum's own bytes). This checksum is stored in the 4 bytes immediately preceding the parent UUID. [*] The dynamic header is the 1024 bytes starting at offset 0x200. There also seems to be a parent filename (tail part of) in unicode form in the dynamic header, 256 16-bit words beginning 8 bytes after the parent UUID. However my own code makes no reference to this field, so I expect that it will normally be all zeros, with filenames supplied by external control files.

Losing a middle member of a snapshot chain is likely to be fatal. I have no doubt that you can eventually fool CloneVDI into building a merged VDI file for you, it should mount in a VM, but there will be massive corruption of the filesystem. It should be a stern test for chkdsk!

[*] If you know C programming then you could also download the CloneVDI source code and temporarily disable the checksum tests in "vhdr.c" (the VHD reader module). I.e. replace all "if (CheckSum(...) == value)" calls with "if (1)".

[WalterC]

Don, you are correct - you were talking about VDI not VHD in the post I was referencing. Sorry for the confusion. I downloaded the VHD spec and used it to determine that the header does contain the unicode parent file name at offset 240-

headerb4.JPG (53.63 KiB) Viewed 6491 times

I edited the file to set to it's new parent file.

headeraft.JPG (70.12 KiB) Viewed 6491 times

Unfortunately I am not a C programmer and I haven't been able to find any sort of tool to calculate a new 1's complement checksum for the updated header. The vboxmanage sethdparentuuid doesn't work on snapshot files. I thought I might trick it into helping me. Looks like I am at a dead end. Foiled again.

Thanks for your assistance!

-Walter

[mpack]

Well, you don't necessarily need a tool. The checksum is quite easy to calculate manually. Just add up all the non-zero bytes in the header, express the value in binary (Windows calculator can do that), and flip all the bits (ones complement). Store in header as four bytes, most sig byte first.

However...

Attached to this message is a modified copy of the current version of CloneVDI. The only change relative to the download on the first page is that this has the VHD checksum test disabled as discussed above.

Note to others: there's no point in anyone but WalterC downloading this, as it's the same as the official download except for what amounts to a deliberate bug.

Walter: let me know when you have this and I'll delete it from here, since I don't want confusion over versions (this still claims to be v2.10).

CloneVDI-exe-2.10-modified.zip

[WalterC]

Success!

I was determined to update the VHD with the correct checksum just so I would make it viable to as many programs as possible. So I became a C programmer for a few hours and used your source and Visual Studio Express. I made a couple of mods in order to determine the checksum you were calculating. I updated the VHD and voila! - clonevdi was happy. I then cloned the entire broken VM using the last snapshot as the source VM into a new VDI using clonevdi. A few hours later I had a working vdi. . No doubt something is missing, but most or all of what I really wanted back was contained in the newer snapshots, and appears to be fine as does Windows (big shock).

Believe me I tried to calculate the checksum manually, but had no luck. I copied all 1024 hex bytes from the header starting at offset 200 into a spreadsheet , zeroed out the 4 existing checksum hex values, converted them to decimal and summed them together. Then I used Windows calculator to do the conversion from decimal to binary, flipped the binary with the NOT button and convert that value back to hex. I tried it on a know good header and it's checksum first and could not get the correct values. So I was obviously doing it wrong.

Thanks again Don for your great tool and willingness to help. Much appreciated!

-Walter

[mpack]

A few hours later I had a working vdi. . No doubt something is missing, but most or all of what I really wanted back was contained in the newer snapshots, and appears to be fine as does Windows (big shock).

Congratulations! I'm impressed that as a non C programmer you were able to get to grips with Visual Studio Express so quickly. Even if you aren't a programmer now, you must have had some prior exposure to development using a compiled language, yes?

I would copy off the files you want first, possibly make a backup, and then I would let chkdsk loose on the disk to see what it can find. That's assuming that you would rather save the VM rather than start from scratch. For future ref, I would recommend that you avoid both VHD and snapshots in future, make proper backups of the VM folder - at least once week and before any significant change.

Believe me I tried to calculate the checksum manually, but had no luck. I copied all 1024 hex bytes from the header starting at offset 200 into a spreadsheet...

You mean, starting at offset 0x200 right? (200 hex, 512 decimal). My earlier message assumed you would be comfortable with hex numbering and notation given the hex editor pics you had already posted.

[WalterC]

Don,

I am not a C programmer, but I am a developer. Mostly mainframe, but some VBA, Access, SQL I did have my struggles with your code and using VS to compile and run it. It kept complaining about a manifest file missing but the hardest part (for me) was getting the value calculated by your CheckSum routine into a variable where I could see it. I gave up trying to get a messagebox type thing working and just used debug.

I'll probably keep using this VM, and will stick with VDI. I have never had an issue like this with Vmware workstation snapshots. I typically take a snapshot when I change something that may break things and if everything works out ok, I don't take the time to merge. The one time I do decide to clean it up, I get bit. Go figure.

Yes definitely using offset 0x200. Right where cxsparse starts. Using the original working copy of the VHD the checksum bytes are FF FF D4 1F. My sum of the 1024 header gives me 11985 decimal. Convert that to binary, flip bits with calculator and convert back to hex gives me D1 2E. Not even close. So it's obvious I don't know what I am doing.

-Walter

[mpack]

I have never had an issue like this with Vmware workstation snapshots.

Well, you may never have encountered the problem with VMWare, but IMHO it simply can't be any safer there, as any engineer can see that the risks are inherent in the concept: if you have a chain of dependencies with little redundancy, then loss or corruption (for any reason) of one of the earlier elements must be passed down the dependency chain to later elements, where the lack of redundancy will tend to ensure that losses are serious.

People tend to think of snapshots as a kind of backup. A dangerous mistake! Snapshots are not backups. A backup is where you create redundancy (i.e. by making copies of data) to ensure that you can't ever lose the data. Snapshots don't add redundancy, on the contrary they expose additional critical data (the snapshot control structure) to potential damage. Snapshots are almost literally akin to a house of cards - exactly the same inherent structural weakness.

[WalterC]

Just to conclude, running chkdsk found no problems whatsoever in the recovered VM and as far as I can tell so far, no data is missing or corrupted.
I don't have any idea how big the missing snapshot file was, but it must not have contained anything important.

Thanks again for the assistance.

-Walter

[mpack]

It looks like you were incredibly lucky.