CloneVDI tool - Discussion & Support

[socratis]

Yeap, that's what I downloaded and looked at. I don't see it, honestly...

<MediaRegistry>
  <HardDisks>
    <HardDisk uuid="{ba3d4ae6-fdc7-4d9e-9ffd-56229ae0fae4}" location="debian-9-test.vdi" format="VDI" type="Normal"/>
  </HardDisks>
  <DVDImages>
    <Image uuid="{7cea98ef-c2d6-45ad-ab5e-7d66dfa042e5}" location="D:/ISOs/Linux/debian-9.0.0-amd64-xfce-CD-1.iso"/>
  </DVDImages>
</MediaRegistry>
...
<StorageControllers>
  <StorageController name="SATA" type="AHCI" PortCount="2" useHostIOCache="true" Bootable="true" IDE0MasterEmulationPort="0" IDE0SlaveEmulationPort="1" IDE1MasterEmulationPort="2" IDE1SlaveEmulationPort="3">
    <AttachedDevice type="HardDisk" hotpluggable="false" port="0" device="0">
      <Image uuid="{ba3d4ae6-fdc7-4d9e-9ffd-56229ae0fae4}"/>
    </AttachedDevice>
    <AttachedDevice passthrough="false" type="DVD" hotpluggable="false" port="1" device="0"/>
  </StorageController>
</StorageControllers>

[mpack]

Looks like you didn't notice my edit.

[socratis]

Nope. It seems your edit was done after I read your original message and replied, or I was in the process of replying...

[mpack]

I have some news on the Debian 9 problem. I have been able to easily reproduce the problem following Beppler's guidance.

It's early days yet, and I don't have a fix, but I suspect the problem has nothing to do with checksuming or with selecting the 64bit variant of the filesystem: all the latter seems to do is use some previously unused/zero fields in the header (superblock) to store the upper 32bits of various global total (total blocks used etc). These extra fields are all zero for drives of moderate sizes such as we've been discussing and hence this feature should be completely compatible with older EXTx code.

What strikes me as more likely to cause a problem is that the Debian 9 installer enables "extents", a feature that AFAIK didn't exist when I implemented CloneVDI's EXTx support. I guess I should have tested the xxx_compat flags in the header, but CloneVDI's compatibility requirements are a bit odd: I don't normally need to parse the filesystem, all I need to do (for compaction purposes) is understand enough to find my way to the block bitmap(s).

I don't fully understand the feature yet, but I have a nasty feeling that if extents are used, then block bitmaps are not, or not as much. Unfortunately I'm having difficulty finding a clear description of extents. The ext4 kernel wiki only makes fleeting mention of them, and mostly continues to talk about block bitmaps.

I wonder why they decided to do this? It's as if someone had asked "what should we do with all these EXT4 optional gimmicks that got implemented years ago and everyone is too scared to enable? Why don't we just turn them on and find out what breaks!". ISTM the reason wasn't disk size: the 32bit variant of EXTx can handle single volumes up to 16TB, and of course Linux is perfectly capable of handling more than one volume.

[mpack]

Hmm. That may not be it either. I've done some digging into what extents are, and they're pretty much what I expected them to be: the same as the NTFS feature I was already familiar with but whose name I forget. Basically instead of running along the allocation bitmap to find all of a file's blocks, the directory entry references an extent structure which has a table which directly lists all the file's blocks (or "clusters" as they're known in Win-world). If the file is above a certain size (too many blocks to fit in one extent table) then the first extent table lists the location of sub extent tables. Few files need the extent tree to grow deeper than that.

However in NTFS there is still an allocation bitmap which can quickly tell you which blocks are in use. Has EXT4 scrapped the bitmap entirely? I have not yet found an explicit mention of this either way. If I run CloneVDI under a debugger and see what happens when it reads the bitmap - the bitmap in the first block group seems ok, thereafter the bitmaps seem empty. It could just be because I'm dealing with a free install of Debian and the upper block groups are genuinely not used yet, but I find that hard to believe when the Debian9 VDI is 2.6GB.

On the other hand, I just tried the same thing on a 2013 Linux Mint 10 install that I keep for reference. Running it in the debugger (during a compaction op) I see that it too is EXT4 with extents enabled, but not 64bit. Compaction works perfectly. So am I wrong? Is it the 64bit option after all? I don't yet see how.

[mpack]

p.s. My earlier speculation that checksumming might be a problem turns out to be nonsense. During compaction CloneVDI has no need to modify any filesystem structures, hence checksum changes don't arise. CloneVDI only reads filesystem structures to find out what blocks are in use so it can discard the ones which are not.

[mpack]

Ah. I may have it. If the 64bit option is enabled then the "block group descriptor entry" structure doubles in size from 32 to 64 bytes (actually the size becomes configurable). The 32 bytes size inherited from EXT3 was hard coded in CloneVDI. One of the things this structure is used for is to obtain the LBA of the allocation bitmap for each 128MB block group, so this would certainly be screwing me up. I'll check out this theory tomorrow am my time (GMT+1).

[mpack]

Confirmed. I have now implemented and tested a fix for the ex4+incompat_64bit compaction compatibility problem. I want to hold off releasing it for a few days tho because there are other changes I've been thinking about making. This problem happened with Debian 9 since the latter now seems to enable the 64bit option by default, I couldn't find any way to bypass this when EXT4 was selected - I had to select EXT3 (with journalling) instead. Other distros will be affected by the same bug if they enable the EXT4 64bit option. In the past this only happened when the drive was larger than 16TB and so was not expected (by me) to require attention in VMs for some time yet.

[mpack]

NEW VERSION RELEASED - v3.00 (see root message for download link).

Please note that I've released a new version of my CloneVDI tool (v3.00). This release has a number of new features and minor bugfixes, including :-

• CloneVDI now has a built in VDI header repair feature which codifies and automates the repairs that I have done for people manually up to now. The release notes has the details, it's important that you read them if you need this feature.
  
• CloneVDI now allows you to use the "Enlarge" feature to shrink a disk, provided that the new size is large enough to hold all the partitions on the drive. This feature is experimental and you should use it at your own risk: the feature takes no account of the possibility that valid data might be stored outside of the MBR partitioned region - GPT partitions, Windows Dynamic Drives and other things may store data in naughty places. On the other hand you always have your original file to fall back on (CloneVDI will never do in place mods to the source file).
  
• [Bugfix and new feature] CloneVDI failed to test EXT4 compatibility flags properly to test for its incompatibility with EXT4's 64bit option. It now does the test, but I also made it compatible with EXT4-64bit anyway. This fixes recently reported problems with Debian 9 default installs.

Enjoy!

[Beppler]

NEW VERSION RELEASED - v3.00 (see root message for download link).

It worked very well.

Thanks for your efforts and this new version!

[mpack]

It worked very well.

I assume you mean the EXT4-64bit fix for Debian 9. Thanks for confirming it.

[mpack]

Help requested :-

The new header repair feature involved the addition of a new dialog. It would be nice to have translations of this dialog for the additional locales that CloneVDI supports.

The text to be translated is as follows :-

The VDI failed header validation checks and would normally be considered unreadable. CloneVDI can attempt to infer a new header but this may not work and could result in total failure or a badly corrupted clone. You should test the clone (if any) thoroughly before relying on it.

Do you still wish to continue?

Cloning a Corrupted Source File.

I need translations of each of the three strings (sentences) above. Etepetete has already kindly provided a German translation (not yet in the release), but I'm missing French and Dutch versions - it would be great if a volunter could provide either of those it. Btw, the translation should ideally be about the same length.

In the meantime CloneVDI gives this dialog in English or German only (and now French also).

[nathan1258]

Is it for windows?

[mpack]

That's what the forum name says ("Windows Hosts"), so yes. But it also runs under Wine on Linux and OS X hosts.

[markfilipak2]

Cross Reference: http://forums.virtualbox.org/viewtopic. ... 75#p399477

Hello, mpack!

Quoted text is from CloneVDI-exe-3.00, release notes.txt

The "Increase partition size" option is only available when the "Increase virtual drive size" option
is selected. If the "Increase partition size" option is selected then CloneVDI will automatically
increase the size of the main partition to fill the new drive size. This feature only works at
present on FAT16, FAT32 and NTFS partitions, however it has side effects which may be useful to
you even with other guest filesystems (see the notes on increasing partition size later on).

Don, I was not aware that a partition's particular filesystem affected the disk's partition table. I assume you mean expanding & formatting the clone's larger partition, no? If so, I assume from the above that, if the partition is to support a Linux filesystem, you recommend using 'gparted' to enlarge the partition's used space instead of using CloneVDI's "Increase partition size" option, correct?

The "Compact drive while copying" checkbox enables a feature whereby the tool identifies unused
blocks(*) in the guest filesystem, and omits those from the clone. This would be most useful if a
large amount of file data has been deleted inside the guest. This saves you from having to run SDelete
or its Linux equivalent before making the clone, to get the same result. This feature only does
something if CloneVDI recognizes the guest filesystem, and at present the only supported filesystems
are NTFS (tested with NT4 and later), EXT2/3/4, FAT16 and FAT32.

Okay, now I'm unsure whether CloneVDI supports Linux filesystem resizing from original VDI to clone VDI.

Disclosure: I plan to use CloneVDI to simply copy-create a larger VDI, then run the clone to see if everything is joyous, and then to use 'gparted' to enlarge the clone's filesystem. I'm providing this post simply to give you feedback on your documentation.

I will be back with the results of using CloneVDI. I've tried VBox's native "Clone" function but it didn't change fixed to dynamic. I've also tried to use 'vboxmanage' and wound up with a dynamic VMDK that 'vboxmanage' would not register with the VBox Manager (presumably because it's VMDK instead of VDI).

Regards,
Mark.