CloneVDI tool - Discussion & Support

[snailseyes]

I am having trouble with the compact function on CloneVDI. Running it on a Windows 8.1 host, to copy and compact a recently installed Debian guest, results in a vdi file that is smaller than anticipated and corrupted such that it will not boot. Well, it boots to grub rescue because the boot directory, among others, is not accessible.

The problem seems to have arisen after updating Virtualbox to 5.1.18. Apparently none of the vdi files from my virtual machines (Debian and Xubuntu) created with 5.1.18 will successfully compact. Whereas, all of the virtual machines (Mint, Peppermint, Debian, PCLinuxOS) started prior to upgrading Virtualbox (i.e. 5.1.8, 5.0.26, 5.0.10) will compact okay. So most of my existing
vitual machines are fine, but any new ones will not compact successfully.

Am I doing something wrong? Has Virtualbox changed something in the way it creates vdi files?

[mpack]

I'm not aware that anything has changed in VirtualBox. In fact if the VM boots to grub rescue that that would imply that the VDI is fine, but the guest OS doesn't like something about how the content changed.

What is the partition type and filesystem used in the guest? Does the VM work if you clone without compacting? (the latter would prove if it's affected by a change in format). Can you show me the MBR of a defective VDI before and after compaction? Does grub give any information on what it doesn't like about the disk?

Are you sure it isn't the guest OS or grub which is different? It's a well known problem that if you clone a VDI and don't use the "Keep UUID" option then grub will fail to boot on some systems because the grub script (on that install) is set to identify the boot drive by UUID.

[snailseyes]

Thank you mpack. To answer your questions.

It is a primary partition, ext4 filesystem. (Debian installer default settings).

Yes, the VM works if I clone without compacting.

I've dd the MBR for the original, uncompacted and compacted vdi. Dunno how to display these on the forum so link to download. Doh, I can't post urls. I'll try again after I've been a member 24 hours. They are identical, I think.

Grub-rescue, well, I can type set and ls but... insmod said "error: invalid arch-independent ELF magic". As far as I can tell, grub cannot read the grub.cfg file. I attached the vdi to another VM, in which GParted showed the partition structure as expected but warned that the file system was unreadable.

No, I'm not sure it isn't the guest OS or grub which is different. But an un-compacted clone works, so the difference only affects compacted clones; and the problem occurs with more than one distro (although Debian and Xubuntu may well share the same idiosyncracy). Actually, further testing of VMs shows there is a problem with Debian Testing, Xubuntu 17.04 and Manjaro 17, but not with Xubuntu 16.04 or VM of Debian Testing that I've had rolling since Dec 2015.

I had thought it might be a UUID issue, but un-compacted clones work with either a new UUID or keeping the old UUID. Whereas compacted clones don't boot either way, I tried both options, keeping the UUID or generating a new one, neither worked.

So, in summary, compacting the vdi seems to be making the file system unreadable in some of my vdi files. More specifically, from installs of more up-to-date Linux distros (i.e. March 2017 onwards).

[snailseyes]

It was quicker to figure out how to export to txt, than how to post a link to the hex files.

MBR of defective vdi as follows:-

MBR - original vdi

Code: Select all   Expand viewCollapse view

Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000  EB 63 90 10 8E D0 BC 00 B0 B8 00 00 8E D8 8E C0  ëc..ŽÐ¼.°¸..ŽØŽÀ
00000010  FB BE 00 7C BF 00 06 B9 00 02 F3 A4 EA 21 06 00  û¾.|¿..¹..ó¤ê!..
00000020  00 BE BE 07 38 04 75 0B 83 C6 10 81 FE FE 07 75  .¾¾.8.u.ƒÆ..þþ.u
00000030  F3 EB 16 B4 02 B0 01 BB 00 7C B2 80 8A 74 01 8B  óë.´.°.».|²€Št.‹
00000040  4C 02 CD 13 EA 00 7C 00 00 EB FE 00 00 00 00 00  L.Í.ê.|..ëþ.....
00000050  00 00 00 00 00 00 00 00 00 00 00 80 01 00 00 00  ...........€....
00000060  00 00 00 00 FF FA 90 90 F6 C2 80 74 05 F6 C2 70  ....ÿú..ö€t.öÂp
00000070  74 02 B2 80 EA 79 7C 00 00 31 C0 8E D8 8E D0 BC  t.²€êy|..1ÀŽØŽÐ¼
00000080  00 20 FB A0 64 7C 3C FF 74 02 88 C2 52 BB 17 04  . û d|<ÿt.ˆÂR»..
00000090  F6 07 03 74 06 BE 88 7D E8 17 01 BE 05 7C B4 41  ö..t.¾ˆ}è..¾.|´A
000000A0  BB AA 55 CD 13 5A 52 72 3D 81 FB 55 AA 75 37 83  »ªUÍ.ZRr=.ûUªu7ƒ
000000B0  E1 01 74 32 31 C0 89 44 04 40 88 44 FF 89 44 02  á.t21À‰D.@ˆDÿ‰D.
000000C0  C7 04 10 00 66 8B 1E 5C 7C 66 89 5C 08 66 8B 1E  Ç...f‹.\|f‰\.f‹.
000000D0  60 7C 66 89 5C 0C C7 44 06 00 70 B4 42 CD 13 72  `|f‰\.ÇD..p´BÍ.r
000000E0  05 BB 00 70 EB 76 B4 08 CD 13 73 0D 5A 84 D2 0F  .».pëv´.Í.s.Z„Ò.
000000F0  83 D0 00 BE 93 7D E9 82 00 66 0F B6 C6 88 64 FF  ƒÐ.¾“}é‚.f.¶Æˆdÿ
00000100  40 66 89 44 04 0F B6 D1 C1 E2 02 88 E8 88 F4 40  @f‰D..¶ÑÁâ.ˆèˆô@
00000110  89 44 08 0F B6 C2 C0 E8 02 66 89 04 66 A1 60 7C  ‰D..¶ÂÀè.f‰.f¡`|
00000120  66 09 C0 75 4E 66 A1 5C 7C 66 31 D2 66 F7 34 88  f.ÀuNf¡\|f1Òf÷4ˆ
00000130  D1 31 D2 66 F7 74 04 3B 44 08 7D 37 FE C1 88 C5  Ñ1Òf÷t.;D.}7þÁˆÅ
00000140  30 C0 C1 E8 02 08 C1 88 D0 5A 88 C6 BB 00 70 8E  0ÀÁè..ÁˆÐZˆÆ».pŽ
00000150  C3 31 DB B8 01 02 CD 13 72 1E 8C C3 60 1E B9 00  Ã1Û¸..Í.r.ŒÃ`.¹.
00000160  01 8E DB 31 F6 BF 00 80 8E C6 FC F3 A5 1F 61 FF  .ŽÛ1ö¿.€ŽÆüó¥.aÿ
00000170  26 5A 7C BE 8E 7D EB 03 BE 9D 7D E8 34 00 BE A2  &Z|¾Ž}ë.¾.}è4.¾¢
00000180  7D E8 2E 00 CD 18 EB FE 47 52 55 42 20 00 47 65  }è..Í.ëþGRUB .Ge
00000190  6F 6D 00 48 61 72 64 20 44 69 73 6B 00 52 65 61  om.Hard Disk.Rea
000001A0  64 00 20 45 72 72 6F 72 0D 0A 00 BB 01 00 B4 0E  d. Error...»..´.
000001B0  CD 10 AC 3C 00 75 F4 C3 EE 8D B1 AA 00 00 80 20  Í.¬<.uôÃî.±ª..€
000001C0  21 00 83 FE FF FF 00 08 00 00 00 F8 3F 02 00 FE  !.ƒþÿÿ.....ø?..þ
000001D0  FF FF 05 FE FF FF FE 07 40 02 02 F0 3F 00 00 00  ÿÿ.þÿÿþ.@..ð?...
000001E0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000001F0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 AA  ..............Uª

MBR - compacted vdi

Code: Select all   Expand viewCollapse view

Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000  EB 63 90 10 8E D0 BC 00 B0 B8 00 00 8E D8 8E C0  ëc..ŽÐ¼.°¸..ŽØŽÀ
00000010  FB BE 00 7C BF 00 06 B9 00 02 F3 A4 EA 21 06 00  û¾.|¿..¹..ó¤ê!..
00000020  00 BE BE 07 38 04 75 0B 83 C6 10 81 FE FE 07 75  .¾¾.8.u.ƒÆ..þþ.u
00000030  F3 EB 16 B4 02 B0 01 BB 00 7C B2 80 8A 74 01 8B  óë.´.°.».|²€Št.‹
00000040  4C 02 CD 13 EA 00 7C 00 00 EB FE 00 00 00 00 00  L.Í.ê.|..ëþ.....
00000050  00 00 00 00 00 00 00 00 00 00 00 80 01 00 00 00  ...........€....
00000060  00 00 00 00 FF FA 90 90 F6 C2 80 74 05 F6 C2 70  ....ÿú..ö€t.öÂp
00000070  74 02 B2 80 EA 79 7C 00 00 31 C0 8E D8 8E D0 BC  t.²€êy|..1ÀŽØŽÐ¼
00000080  00 20 FB A0 64 7C 3C FF 74 02 88 C2 52 BB 17 04  . û d|<ÿt.ˆÂR»..
00000090  F6 07 03 74 06 BE 88 7D E8 17 01 BE 05 7C B4 41  ö..t.¾ˆ}è..¾.|´A
000000A0  BB AA 55 CD 13 5A 52 72 3D 81 FB 55 AA 75 37 83  »ªUÍ.ZRr=.ûUªu7ƒ
000000B0  E1 01 74 32 31 C0 89 44 04 40 88 44 FF 89 44 02  á.t21À‰D.@ˆDÿ‰D.
000000C0  C7 04 10 00 66 8B 1E 5C 7C 66 89 5C 08 66 8B 1E  Ç...f‹.\|f‰\.f‹.
000000D0  60 7C 66 89 5C 0C C7 44 06 00 70 B4 42 CD 13 72  `|f‰\.ÇD..p´BÍ.r
000000E0  05 BB 00 70 EB 76 B4 08 CD 13 73 0D 5A 84 D2 0F  .».pëv´.Í.s.Z„Ò.
000000F0  83 D0 00 BE 93 7D E9 82 00 66 0F B6 C6 88 64 FF  ƒÐ.¾“}é‚.f.¶Æˆdÿ
00000100  40 66 89 44 04 0F B6 D1 C1 E2 02 88 E8 88 F4 40  @f‰D..¶ÑÁâ.ˆèˆô@
00000110  89 44 08 0F B6 C2 C0 E8 02 66 89 04 66 A1 60 7C  ‰D..¶ÂÀè.f‰.f¡`|
00000120  66 09 C0 75 4E 66 A1 5C 7C 66 31 D2 66 F7 34 88  f.ÀuNf¡\|f1Òf÷4ˆ
00000130  D1 31 D2 66 F7 74 04 3B 44 08 7D 37 FE C1 88 C5  Ñ1Òf÷t.;D.}7þÁˆÅ
00000140  30 C0 C1 E8 02 08 C1 88 D0 5A 88 C6 BB 00 70 8E  0ÀÁè..ÁˆÐZˆÆ».pŽ
00000150  C3 31 DB B8 01 02 CD 13 72 1E 8C C3 60 1E B9 00  Ã1Û¸..Í.r.ŒÃ`.¹.
00000160  01 8E DB 31 F6 BF 00 80 8E C6 FC F3 A5 1F 61 FF  .ŽÛ1ö¿.€ŽÆüó¥.aÿ
00000170  26 5A 7C BE 8E 7D EB 03 BE 9D 7D E8 34 00 BE A2  &Z|¾Ž}ë.¾.}è4.¾¢
00000180  7D E8 2E 00 CD 18 EB FE 47 52 55 42 20 00 47 65  }è..Í.ëþGRUB .Ge
00000190  6F 6D 00 48 61 72 64 20 44 69 73 6B 00 52 65 61  om.Hard Disk.Rea
000001A0  64 00 20 45 72 72 6F 72 0D 0A 00 BB 01 00 B4 0E  d. Error...»..´.
000001B0  CD 10 AC 3C 00 75 F4 C3 EE 8D B1 AA 00 00 80 20  Í.¬<.uôÃî.±ª..€
000001C0  21 00 83 FE FF FF 00 08 00 00 00 F8 3F 02 00 FE  !.ƒþÿÿ.....ø?..þ
000001D0  FF FF 05 FE FF FF FE 07 40 02 02 F0 3F 00 00 00  ÿÿ.þÿÿþ.@..ð?...
000001E0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000001F0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 AA  ..............Uª

MBR- uncompacted vdi

Code: Select all   Expand viewCollapse view

Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000  EB 63 90 10 8E D0 BC 00 B0 B8 00 00 8E D8 8E C0  ëc..ŽÐ¼.°¸..ŽØŽÀ
00000010  FB BE 00 7C BF 00 06 B9 00 02 F3 A4 EA 21 06 00  û¾.|¿..¹..ó¤ê!..
00000020  00 BE BE 07 38 04 75 0B 83 C6 10 81 FE FE 07 75  .¾¾.8.u.ƒÆ..þþ.u
00000030  F3 EB 16 B4 02 B0 01 BB 00 7C B2 80 8A 74 01 8B  óë.´.°.».|²€Št.‹
00000040  4C 02 CD 13 EA 00 7C 00 00 EB FE 00 00 00 00 00  L.Í.ê.|..ëþ.....
00000050  00 00 00 00 00 00 00 00 00 00 00 80 01 00 00 00  ...........€....
00000060  00 00 00 00 FF FA 90 90 F6 C2 80 74 05 F6 C2 70  ....ÿú..ö€t.öÂp
00000070  74 02 B2 80 EA 79 7C 00 00 31 C0 8E D8 8E D0 BC  t.²€êy|..1ÀŽØŽÐ¼
00000080  00 20 FB A0 64 7C 3C FF 74 02 88 C2 52 BB 17 04  . û d|<ÿt.ˆÂR»..
00000090  F6 07 03 74 06 BE 88 7D E8 17 01 BE 05 7C B4 41  ö..t.¾ˆ}è..¾.|´A
000000A0  BB AA 55 CD 13 5A 52 72 3D 81 FB 55 AA 75 37 83  »ªUÍ.ZRr=.ûUªu7ƒ
000000B0  E1 01 74 32 31 C0 89 44 04 40 88 44 FF 89 44 02  á.t21À‰D.@ˆDÿ‰D.
000000C0  C7 04 10 00 66 8B 1E 5C 7C 66 89 5C 08 66 8B 1E  Ç...f‹.\|f‰\.f‹.
000000D0  60 7C 66 89 5C 0C C7 44 06 00 70 B4 42 CD 13 72  `|f‰\.ÇD..p´BÍ.r
000000E0  05 BB 00 70 EB 76 B4 08 CD 13 73 0D 5A 84 D2 0F  .».pëv´.Í.s.Z„Ò.
000000F0  83 D0 00 BE 93 7D E9 82 00 66 0F B6 C6 88 64 FF  ƒÐ.¾“}é‚.f.¶Æˆdÿ
00000100  40 66 89 44 04 0F B6 D1 C1 E2 02 88 E8 88 F4 40  @f‰D..¶ÑÁâ.ˆèˆô@
00000110  89 44 08 0F B6 C2 C0 E8 02 66 89 04 66 A1 60 7C  ‰D..¶ÂÀè.f‰.f¡`|
00000120  66 09 C0 75 4E 66 A1 5C 7C 66 31 D2 66 F7 34 88  f.ÀuNf¡\|f1Òf÷4ˆ
00000130  D1 31 D2 66 F7 74 04 3B 44 08 7D 37 FE C1 88 C5  Ñ1Òf÷t.;D.}7þÁˆÅ
00000140  30 C0 C1 E8 02 08 C1 88 D0 5A 88 C6 BB 00 70 8E  0ÀÁè..ÁˆÐZˆÆ».pŽ
00000150  C3 31 DB B8 01 02 CD 13 72 1E 8C C3 60 1E B9 00  Ã1Û¸..Í.r.ŒÃ`.¹.
00000160  01 8E DB 31 F6 BF 00 80 8E C6 FC F3 A5 1F 61 FF  .ŽÛ1ö¿.€ŽÆüó¥.aÿ
00000170  26 5A 7C BE 8E 7D EB 03 BE 9D 7D E8 34 00 BE A2  &Z|¾Ž}ë.¾.}è4.¾¢
00000180  7D E8 2E 00 CD 18 EB FE 47 52 55 42 20 00 47 65  }è..Í.ëþGRUB .Ge
00000190  6F 6D 00 48 61 72 64 20 44 69 73 6B 00 52 65 61  om.Hard Disk.Rea
000001A0  64 00 20 45 72 72 6F 72 0D 0A 00 BB 01 00 B4 0E  d. Error...»..´.
000001B0  CD 10 AC 3C 00 75 F4 C3 EE 8D B1 AA 00 00 80 20  Í.¬<.uôÃî.±ª..€
000001C0  21 00 83 FE FF FF 00 08 00 00 00 F8 3F 02 00 FE  !.ƒþÿÿ.....ø?..þ
000001D0  FF FF 05 FE FF FF FE 07 40 02 02 F0 3F 00 00 00  ÿÿ.þÿÿþ.@..ð?...
000001E0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000001F0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 AA  ..............Uª


[mpack]

Hi. A bit of a misunderstanding there: I don't have time to spend manually parsing the raw hex of an MBR, I wanted you to use the "Partition Info" button in CloneVDI, which shows the important info from the MBR, already parsed.

Ok, we seem to have established that the UUID change is not the issue. I have in mind another test: could you possibly create a new VM using latest VirtualBox and an older OS that you know previously worked? Ideally using the same ISO installer that you used originally. This should prove whether the problem is a change in VirtualBox or a change in the guest OS.

[snailseyes]

Yes, I can do that. Creating a new VM using my latest Virtualbox (5.1.18) and Mint 17.3 previously installed and compacted successfully. This new vdi compacts successfully. Virtualbox works. So the problem originates with the newer guest OS. Different kernels? Changes to ext4 filesytem?

I can't see an easy way to export the Partition Info button output. Snipping Tool it is.

VDI Original.png (8.96 KiB) Viewed 5918 times

[mpack]

Thanks for the screenshots: I replaced your offsite links with local attachments, as that's what we prefer around here where possible.

There seems to be no difference in the partition maps, which means that CloneVDI compaction is not discarding the first block which contains the MBR (a genuine bug that CloneVDI once had, though it was fixed years ago). So, it isn't an error in the VDI format, CloneVDI doesn't seem to be throwing away data that it shouldn't, I'm running out of possible error mechanisms.

The only time I've seen this sort of thing happen before is when guest code is using unpartitioned areas to hide data. Unpartitioned areas containing data gets flushed and discarded when compaction is enabled. Surely an open source OS wouldn't do that. Has grub perhaps grown tired of being confined to the first 1MB of the drive? If their solution was to store data in unpartioned upper reaches then those programmers need to be removed from the project: there's a reason why disks have partitions, i.e. why they have markers showing which parts of the drive are in use.

You might want to ask about filesystem changes on a relevant forum, with specific reference to a "smart" disk cloning scenario (one which ignores unused areas of the disk - common in Acronis, Macrium Relect etc). If you identify the exact reason for the boot failure then we can be much clearer about what's needed to fix it.

[snailseyes]

Thanks for taking the time on this.

When I tried to fsck the ext4 partition of the problem vdi, an error message was thrown up complaining about metadata_csum (and saying I needed a more up to date version of e2fsprogs). Googling metadata_csum throws up references to some Linux distros having changed the default flags for ext4, i.e. metadata_csum and 64bit. https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1601997 . Apparently this has caused some problems with backward compatability.

Using an updated fsck, the filesystem of the compacted vdi seems to be badly corrupted. Not so much an intact OS with a broken boot, but an intact boot with a missing OS.

Could the changes to metadata_csum and 64bit options for ext4 be affecting CloneVDI's 'compact' algorithms?

[mpack]

I'm not sure what "64bit options for ext4" might mean. A filesystem specification doesn't change just because a 64bit (or any other Xbit) OS is accessing it.

However, "metadata_csum", which I assume means "metadata checksum", sounds very much like the source of the problem. I assume it's intended to detect tampering with (or corruption of) the filesystem, which of course I do - that being the whole point of the compaction feature. There would be no actual corruption, it's just that the checksum is no longer correct.

I don't actually remember a checksum option in EXTx, but it has been a long time since I wrote the code - and perhaps this was added since then but left disabled until very recently. In fact ISTR I wrote the code for EXT3, then when EXT4 first appeared I found that changes were minimal. In that case it would require a new version of CloneVDI to support this change, and I'm afraid that my time is quite congested for the near future.

To prove this you could turn off the metadata_csum option on the partition(s).

To disable checksums on an existing filesystem, ensure that the filesystem will pass fsck. Then turn off metadata_csum via tune2fs, e.g. tune2fs -O ^metadata_csum /dev/path/to/disk.

I suspect you may then be able to compact the filesystem wihout complaint, if this is the only change to the filesystem structures.

[Rodger_Moore]

I'm not sure what "64bit options for ext4" might mean. A filesystem specification doesn't change just because a 64bit (or any other Xbit) OS is accessing it.

However, "metadata_csum", which I assume means "metadata checksum", sounds very much like the source of the problem. I assume it's intended to detect tampering with (or corruption of) the filesystem, which of course I do - that being the whole point of the compaction feature. There would be no actual corruption, it's just that the checksum is no longer correct.

I don't actually remember a checksum option in EXTx, but it has been a long time since I wrote the code - and perhaps this was added since then but left disabled until very recently. In fact ISTR I wrote the code for EXT3, then when EXT4 first appeared I found that changes were minimal. In that case it would require a new version of CloneVDI to support this change, and I'm afraid that my time is quite congested for the near future.

To prove this you could turn off the metadata_csum option on the partition(s).

To disable checksums on an existing filesystem, ensure that the filesystem will pass fsck. Then turn off metadata_csum via tune2fs, e.g. tune2fs -O ^metadata_csum /dev/path/to/disk.
I suspect you may then be able to compact the filesystem wihout complaint, if this is the only change to the filesystem structures.

I'm running into the exact same problems on a Ubuntu 14.04.5_LTS server (Linux Kernel 4) guest system. Getting a fatal error after grub trying to boot to Ubuntu on the cloned disk. I really want to compact the disk because I loaded a large OpenStreetMap dataset into Postgres. It uses a bunch of temp tables so disk size explodes doing this. After loading the data it deletes the temp tables but the vdi disk is still huge. I hope turning off metadata_csum on the partitions solves this. I'm kinda confused though because I have a LVM setup. So should I run the tune2fs -O ^metadata_csum on /dev/mapper/server---lv--blablabla or on /dev/sdax ??

Thanks!

EDIT: I tried this but it didn't work out. I guess it has something to do with LVM...?

[mpack]

I believe it's mentioned in the CloneVDI docs that LVM is not supported. Only MBR type partition maps and simple stand alone drives.

But if you're using LVM then it shouldn't have been possible to get the same compaction problem as Snailseyes: if you use LVM then CloneVDI can't compact it, period. You have to fall back on the old zerofill techniques.

[socratis]

Thanks for the quick reply. Clear!

[Beppler]

Hi.

First thanks for this awesome tool.

I have been using it for a while and had success cloning and compacting many images (debian 8, ubuntu 14 and 16, centos 6 and 7).

But when I try compact an image with a debian 9 the image is being generated with less blocks than the necessary.

After doing a minimal install with only one primary partition for root using ext4 (after the install a swap file is created and dkms is installed) the vdi file has 1553 blocks allocated, after clone with compact the new vdi has only 323 blocks allocated and the image is unbootable.

Using df on original image reports that there are 1472828 of 16446332 blocks used.

[mpack]

What partitioning scheme and filesystem does Debian 9 use? If there's a choice then say what you chose.

[Beppler]

I created an primary partition using all the disk and used ext4 as filesystem mounted as "/".