CloneVDI tool - Discussion & Support

Discussions related to using VirtualBox on Windows hosts.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

SoftExpert wrote:You might be right ... but how to explain the fact that the same gparted tool can copy and resize the same source partition on an new VDI without creating the issue? Something has to be different between the two methods ... could it be that the partition table is modified in such a way that gparted does not see all the records?
Sorry, I consider myself an expert on what CloneVDI does, but not an expert on what gparted does, so I can't answer questions about the latter. I'll repeat however that CloneVDI does not modify the partition table in any way. In fact this is easily verified: CloneVDI has a dialog box which shows you the partition table, simply open the "before CloneVDI" and "after CloneVDI (but before gparted)" VDIs, and view the partition table on each. You will see that they are the same. Let me know if they are not! :-)
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

SoftExpert wrote:but how to explain the fact that the same gparted tool can copy and resize the same source partition on an new VDI without creating the issue?
Actually, I've just realized what the problem must be, and the problem is precisely that I do not modify the partition table. The other unexpected factor is that the drive sizes you have chosen are strangely small by modern standards. To quote my own release notes: "nobody is immune to bugs, and it's quite possible that if you try something unusual then you'll hit a problem". :)

To explain: each partition table entry in the MBR has the start and end address of the partition expressed in two formats, the old DOS style CHS (cylinder,head,sector) addressing format, and the later LBA (logical block address) format. The old CHS format had a limited address range: maximums for C,H,S were 1023,255,63. If you multiply this out you get 16434495 sectors, so this is the maximum sector number you can express in CHS. In bytes it's just short of 8GB. Above 8GB the CHS fields are maxed out and only the LBA fields matter. The LBA fields can express sector addresses up to 2TB.

Virtual drives simulate a drive geometry, and VirtualBox calculates what geometry to simulate based on the drive size. Above 8GB and the geometry has maxed out and doesn't make a difference anymore, but below 8GB it does matter! What it means is that if I enlarge a drive which starts off smaller than 8GB then I need to calculate the new drive geometry and patch the partition start and end CHS addresses to suit.

To prove this, you could try repeating your experiment, but this time start by creating a drive with a virtual size of 16GB, then use your procedure to expand this to 32GB. You should find that Windows no longer reports any inconsistent statistics.

This is a bit of a surprise, since I thought modern operating systems by now ignored the legacy CHS fields altogether, unless the LBA fields in the MBR are zero.

Another thing I'm not sure about is how well gparted can cope with very small drives. I'm surprised that it seemed to successfully resize the partition without writing a new max CHS.

In the meantime, do NOT delete your old VDI, because the resized one is now suspect.

I'll investigate this problem a bit more tomorrow.
Etepetete
Posts: 400
Joined: 7. Oct 2009, 10:19
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Slackware 14.2
Location: Berlin

Re: CloneVDI tool - Discussion & Support

Post by Etepetete »

I would think the problem lies with gparted-live-0.4.7-1.iso. This is a testing and not a stable version. As I posted a bit earlier in this thread, I lost my MBR and partition tables when I partitioned my hard drive. I used gparted-live-0.4.6-1. I had to go back to gparted-live-0.4.5-2 to get my partitions done right. If you read the bug reports from gparted, you will notice that they have discovered a bug in gparted-live-0.4.6-1 and if I remember correctly, it had something to do with SATA drives.
eFloh
Posts: 45
Joined: 18. Jan 2009, 17:42
Primary OS: MS Windows Vista
VBox Version: PUEL
Guest OSses: Windows, Debian Linux, LiveCd-Tests

Re: CloneVDI tool - Discussion & Support

Post by eFloh »

@SoftExpert: What happens when you do a fsck ("scandisk") on the partition after resizing? does this fix the problem?
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

Well, despite my fine theory that I knew what the problem was, I've been so far unable to reproduce it.

I created a new VDI of 4GB, and used XP (Pro, 2002 version, SP2) disk manager to partition then format the drive with an NTFS filesystem. I then shut down the VM and used CloneVDI to check the geometry (from the VDI header) and the partition map. The geometry was CHS=(522,255,63), the partition start was at CHS=(0,1,1) - as it always is - and the partition end was at CHS=(521,254,63).

I then used CloneVDI to enlarge the drive to 8GB, then I started the VM (this allows VirtualBox to modify the drive geometry in the header) and shut it down again (so I could access it again with CloneVDI). CloneVDI now reported the drive geometry as CHS=(1024,255,63), the partition end was still at CHS=(521,254,63). So far this is fine: the number of heads and sectors per track didn't change, only the number of cylinders changed, which means that the partition end CHS is still valid (it just means that the cylinder number of the partition end is no longer the last cylinder on the drive, and that is obviously ok).

I then booted up the VM with gparted livecd 0.4.6.1, resized the partition to fill the 8GB, and shut down the VM again (i.e. at this point XP has not seen the enlarged partition). Again checking the partition map with CloneVDI I found that the partition end address was now at CHS=(1023,254,63), which means it has maxed out (as I mentioned before, the max CHS address is just short of 8GB).

I booted up XP, let it do the chkdsk check that it always wants to on boot after gparted has been used, and then looked at the drive in Disk Manager and Explorer disk properties. Both showed the drive as 8GB, meaning I was unable to reproduce the first bug report.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

@SoftExpert:

Despite my lack of results, it's still possible that my guess at the cause was correct. My testing was obviously done with the same version of VirtualBox throughout (v3.0.6), it's possible however that your 4GB VDI was created with an older version of VirtualBox which treated the drive geometry in a different way. Could you please report some of the numbers that I got in my last test? In particular:

Using CloneVDI on the original (unmodified) 4GB drive I want drive geometry (from the bottom of the VDI header dialog), from the same dialog I need the number of blocks (total and allocated). I would also like the CHS addresses of the first and last sectors of any non-zero partition on that drive (from the partition map button, and I expect there is only one non-zero partition).

I would then like you to enlarge the VDI as you did before, start up and shut down the VM using the new VDI but without doing anything else, then give me the drive geometry, and the partition begin and end CHS values for the enlarged VDI.

Finally, use gparted to enlarge the partition in the new VDI, shut down and get the same stats again. Also confirm that you get the same inconsistency in Windows that you got before.
Last edited by mpack on 18. Oct 2009, 14:17, edited 1 time in total.
Etepetete
Posts: 400
Joined: 7. Oct 2009, 10:19
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Slackware 14.2
Location: Berlin

Re: CloneVDI tool - Discussion & Support

Post by Etepetete »

Hi mpack, following is the result of the conversion of a raw image using the VBoxManage convertfromraw command:

So, the situation is the same as when I used your tool, i.e. the image is located on an external HD connected via usb. The file was read from and written to the external HD.
The VBoxManage convertfromraw command took just under 3 hours to do the conversion. This corresponds to the first time I used it although then, the file was read from an external HD and written to the local HD.

Your tool was created for the purpose of cloning virtual disks. That it does, very well. The fact that it can convert raw images is an extra capability, which (as far as I understand) wasn't a goal to begin with. Yes, your tool took much longer in my test, which in my opinion doesn't reduce its value when considering the extras which your tool provides that are not available to the VBoxManage convertfromraw command such as progress bar, the ability to compact the drive while converting or to increase its size all in one go (and others). I also believe it will appeal to a lot of people, who prefer (or cannot do without) a GUI. Most important though, even if it did take more than 100% longer to complete, it is dependable. I am now using the VDI which I converted using your tool.

As stated in a prior post, my notebook doesn't have the resources to test your tool converting from internal HD to internal HD. I'll leave that to someone else.

If my opinion is worth anything, your tool is great! Thank you for allowing me to take part in your project. I learned a hell of a lot and am grateful for it.
Keep up the good work.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

Hi Etepatete, thanks again for your feedback, which has been very good, and thank you for the compliments.
Etepetete wrote:Yes, your tool took much longer in my test
Well, as I mentioned before, the poor performance you saw will have been because you were using a USB drive. To make matters worse you were both reading and writing from USB, which of course at least doubles the USB overhead (perhaps more than doubles, depending on exactly how your PC is set up). If your test had been with a USB flash drive then it would have been far worse: I'm afraid there's not much I can do to improve the performance of the physical hardware!

I don't understand how the VBoxManage performance from USB to USB can be so much better than mine. That doesn't gel with the normal performance comparison between CloneVDI and VBoxManage... unless there's some clever way to use a USB device that the VBox devs are aware of but I'm not.
Etepetete wrote:Thank you for allowing me to take part in your project. I learned a hell of a lot and am grateful for it.
VirtualBox has been a wonderful educational tool for me too. I consider myself an experienced programmer, but... You will see from my profile that I started using VBox a little more than a year ago. Before then I had maybe gotten the chance to install an operating system maybe once every 3 years or so. Exposure to setting up a network? None. Exposure to Linux? None. Since then I've learned a vast amount about installing operating systems of several flavours, how to get networking connections going, the nature and virtues of various virtual disk formats, ditto for various guest filesystems... and probably lots of other things too. When I started out VBox was just a curiosity, I had no idea it would be this much fun! :-)
SoftExpert
Posts: 13
Joined: 16. Oct 2009, 16:53
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: WinXP, Win2K3

Re: CloneVDI tool - Discussion & Support

Post by SoftExpert »

@mpack
mpack wrote:@SoftExpert:

Despite my lack of results, it's still possible that my guess at the cause was correct. My testing was obviously done with the same version of VirtualBox throughout (v3.0.6), it's possible however that your 4GB VDI was created with an older version of VirtualBox which treated the drive geometry in a different way. Could you please report some of the numbers that I got in my last test? In particular:

Using CloneVDI on the original (unmodified) 4GB drive I want drive geometry (from the bottom of the VDI header dialog), from the same dialog I need the number of blocks (total and allocated). I would also like the CHS addresses of the first and last sectors of any non-zero partition on that drive (from the partition map button, and I expect there is only one non-zero partition).

I would then like you to enlarge the VDI as you did before, start up and shut down the VM using the new VDI but without doing anything else, then give me the drive geometry, and the partition begin and end CHS values for the enlarged VDI.

Finally, use gparted to enlarge the partition in the new VDI, shut down and get the same stats again. Also confirm that you get the same inconsistency in Windows that you got before.
Hello mpack (and sorry for my silence - girlfriend involved ;-) ...)
I have done the test with the 3.0.8 version of VirtualBox:
1. created from scratch a new VDI of 256 Mb in size
2. Copied some files on it
3. Resized VDI with CloneVDI 1.40
5. Resized partition with gparted
6. Mounted on Windows : the size miscalculation/inconsistency is there!

Screenshots:
1. Original VDI (256 Mb in size)
testCloneVDI.png
testCloneVDI.png (123.71 KiB) Viewed 12330 times
2. Resized VDI (1 Gb in size)
testCloneVDI_resized2.png
testCloneVDI_resized2.png (73.79 KiB) Viewed 12330 times
I can send you (or make available somewhere) a RAR file with the two VDI files, if you need ...

Best regards,
SoftExpert
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

This is very strange. I set out to repeat your experiment, starting with a 256MB drive. I created the drive, ran the XP VM. I used XP's disk manager to partition then format the drive with NTFS. I copied some files on just to repeat what you did, but I don't think that step matters.

I then closed the VM and looked at the drive geometry and partition map with CloneVDI. The drive geometry was given as CHS=(520,16,63), and the address of the last sector of the partition was given in the partition map as CHS=(31,254,63). ...

But hang on a mo, I haven't done anything with CloneVDI or gparted yet, and isn't that last sector address total bs?! It has an H value that is more than the number of heads given in the drive geometry! By my reckoning the last sector CHS should have been (509,15,63) (XP had made the partition 514017 sectors long). Even if you allow that the geometry can be exceeded, when I convert (31,254,63) into an LBA (see wikipedia for the formula), I can't make it give the expected LBA for the last sector of the partition.

This puts the problem into the lap of XP, or maybe VBox isn't reporting the drive geometry properly. It's hard to believe that XP could screw up something as fundamental to an OS as disk management... Ok, XP won't meet many 256MB local drives in its life, but 256MB external drives used to be quite common (it was a standard USB flash drive size).
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

I just repeated the experiment, creating a 256MB VDI, but this time I mounted it in a FreeDOS VM and used fdisk to partition the drive. This time the partition end CHS was exactly what I expected it should be. That means that VBox reported the drive geometry in a form that FreeDOS at least was happy with, which seems to point the finger at XP.

I can add code to CloneVDI to check and fix the primary MBR, but this doesn't seem to be a CloneVDI problem. It begins to look like XP itself doesn't partition <8GB drives correctly.
SoftExpert
Posts: 13
Joined: 16. Oct 2009, 16:53
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: WinXP, Win2K3

Re: CloneVDI tool - Discussion & Support

Post by SoftExpert »

mpack wrote:I just repeated the experiment, creating a 256MB VDI, but this time I mounted it in a FreeDOS VM and used fdisk to partition the drive. This time the partition end CHS was exactly what I expected it should be. That means that VBox reported the drive geometry in a form that FreeDOS at least was happy with, which seems to point the finger at XP.

I can add code to CloneVDI to check and fix the primary MBR, but this doesn't seem to be a CloneVDI problem. It begins to look like XP itself doesn't partition <8GB drives correctly.
I just saw that the "Legacy geometry" has the same values between the two VDIs. Is this normal?
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

Legacy geometry doesn't matter, just the drive geometry at the bottom of the "VDI header" dialog box - it's "legacy" as in "a header field no longer used by VBox".
Etepetete
Posts: 400
Joined: 7. Oct 2009, 10:19
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Slackware 14.2
Location: Berlin

Re: CloneVDI tool - Discussion & Support

Post by Etepetete »

Regarding performance of your utility while using external HD over USB:

It might increase performance and reduce conversion time, if you buffered writng to HD a bit more generously, i.e. while using the VBoxManage convertfromraw command, I observed the VDI in a windows explorer window and watched the VDI file size grow and how the creation date and the file size changed. But once the actual data had been written (ca. 20 GB) and only empty space was left to be converted (ca. 130 GB), the file size and creation date only changed once an hour. It appears that empty space was written to the external HD only after converting ca. 65 GB free space.
Sasquatch
Volunteer
Posts: 17798
Joined: 17. Mar 2008, 13:41
Primary OS: Debian other
VBox Version: PUEL
Guest OSses: Windows XP, Windows 7, Linux
Location: /dev/random

Re: CloneVDI tool - Discussion & Support

Post by Sasquatch »

Concerning the performance on USB drives, I cloned a few VDIs myself which are stored on an NTFS formatted external drive to that same external hard drive on Linux using Wine. When a lot of data is read and/or written on an NTFS drive in Linux, the CPU goes up a lot until it's all done. Wine adds another hit on performance. But even with all that, my 30 GB VDI (which was about 3 GB filled and enlarged to almost 4 GB) did a fine job in decent time. Transfer speed was about 10 MB/s, not all that bad, 10 MB/s read and 10 MB/s write. All that through one USB cable. Not too shabby.
Read the Forum Posting Guide before opening a topic.
VirtualBox FAQ: Check this before asking questions.
Online User Manual: A must read if you want to know what we're talking about.
Howto: Install Linux Guest Additions
Howto: Use Shared Folders on Linux Guest
See the Tutorials and FAQ section at the top of the Forum for more guides.
Try searching the forums first with Google and add the site filter for this forum.
E.g. install guest additions site:forums.virtualbox.org

Retired from this Forum since OSSO introduction.
Post Reply