CloneVDI tool - Discussion & Support

[Etepetete]

Thanks a lot! Appreciate your help. May I suggest allowing any extension, even no extension? The reason being that dd allows naming the image any way a user decides. If I understand correctly, if the internals of the file don't meet expectations of your program, it won't do anything. Correct?

[mpack]

Thanks a lot! Appreciate your help. May I suggest allowing any extension, even no extension? The reason being that dd allows naming the image any way a user decides.

Well, you are certainly free to suggest it, but at the moment I'm not inclined to agree!

A raw file does not have any header I can use to verify that it is what it should be, so the file extension provides a discipline on the users part to do that check. If I treated any unrecognized extension as meaning a raw image then it means that a newbie could accidentally pick any old file, and the tool would look like it did something meaningful with it, perhaps wasting a lot of my time and his/hers.

If I understand correctly, if the internals of the file don't meet expectations of your program, it won't do anything. Correct?

With any of the other disk formats that would be true, because there's a header in those formats that I can verify. Not so with .RAW. Yes, I do inspect the file, and yes, there are certain additional features that are turned on if I do happen to recognize the contents, but I don't reject the file if I don't recognize it. I emphasise that I'm talking now about how it handles .RAW files - all other disk formats have headers which allow stronger verification.

In any case it surely isn't a big deal to rename the file, regardless of what you called it when you created it.

[Etepetete]

I have finally got around to testing your tool. I've read your comment in that other thread and am sorry if you got the impression I left you hanging. If you took a look at the time I posted my success story you will notice that it was in the wee hours of the night (or morning) when I posted. I've been working on getting VB to work for about 3 weeks now. One week I was ill and could only sit in front of the computer a couple hours at a time. (It's no fun researching on the net when you have to keep the blobs of snot from falling onto your keyboard!) VB wasn't causing the delay, after partitioning my HD I lost my MBR and partition tables. Believe it or not, it happened a couple of times. I had to go back to a relatively old version of GParted before I got satisfactory results. In addition to that I had to go out and acquire a new external HD. I just plain ran out of space to store backups.

So, now that both our egos have been stroked, lets get to the subject of cloning (converting) raw files to VDI. The VDI I am using was created using the VBoxManage convertfromraw command. It evidently worked. I now have a VDI with a virtual size of 150 GB and a literal size of 20 GB. It took t least three hours to convert. The image was stored on an external HD connected via USB.

I changed the file extension of my image to .raw and your tool found it no problem. Your tool refuses to convert it though. The raw image is literally 150 GB in size. The Partition to which I want to save the clone to is only 90 GB in size. Your tool says there is not enough space available to save it where I choose. The VBoxManage command is more tolerant on this point.

Once my new external HD is finally formatted I will try again, saving it on a partition with ca. 500 GB of free space. Then I will report to you how fast it worked and whether the VDI was usable.

[mpack]

Thanks for reporting back. The point however is not about stroking egos, it's about building up a knowledge base in this forum so that future users can learn from earlier experiences, and gain confidence that something they are trying ought to work.

I changed the file extension of my image to .raw and your tool found it no problem. Your tool refuses to convert it though. The raw image is literally 150 GB in size. The Partition to which I want to save the clone to is only 90 GB in size. Your tool says there is not enough space available to save it where I choose.

Yep, my tool make a conservative assumption that the clone will need about the same disk space as the original, which is why it refuses to start the process.

I now have a VDI with a virtual size of 150 GB and a literal size of 20 GB.

I'm surprised VBoxManage was that effective at reducing the size, since it can only discard blocks if they are zeroed (it doesn't understand guest filesystems). Did you run sdelete first perhaps?

Once my new external HD is finally formatted I will try again, saving it on a partition with ca. 500 GB of free space. Then I will report to you how fast it worked and whether the VDI was usable.

Well, if the external HDD is on a USB port then you can't expect fantastic speed, but I'll look forward to your report in any case.

[Etepetete]

I used Eraser to zero out empty space. HD is still being formatted. Report will follow.

Update:Formatting took two hours. Copying the raw image file from one external HD to the new one took about the same time. At the moment I am converting from raw using your tool. Uptime 4 Hr 5 min, time remaining 4 Hr 8 min.

Don't despair, The time factor is not comparable to my previous conversion using the VBoxManage createfromraw command as then I converted from external HD to local HD. (I also didn't time it accurately as there was no reason for me to do it at the time.) Now I am converting from external HD to same external HD. I will work on testing the rest of today and part of tomorrow. Just so you can start drooling, here are the tests I have planned:

1. convert raw image file to VDI without compacting (in Progress).
2. Test if it works in VB version 3.0.8-53140
3. convert raw image file to VDI with compacting.
4. Test if it works in VB version 3.0.8-53140
5. convert raw image file to VDI using VBoxManage createfromraw command (this VDI I will not test as I have successfully done this already)

After completion of the tests, I hope to be able to give you comparable results. (I hope I have enough time to do this by midday tomorrow, a day only has twenty four hours.) If you're interested, I'll post intermediate results.

[mpack]

Ah - sorry for not commenting on your progress, I hadn't noticed that the message had been edited.

On the test with and without compacting: actually there's no point in that, because the compacting is done by an intermediate level layer that understands filesystems but knows nothing about the underlying disk format: it is just given an encapsulated data source object and it calls the method to read the data. In other words the same compaction code runs regardless of whether the source is VDI, VHD, VMDK or RAW. And we know that the data read works, since cloning without compacting would also fail in that case.

The time estimate is likely to be WAY off (i.e. it will finish much faster), since like the earlier check for disk space, the time to completion starts off with the assumption that it'll really have to copy all that data. However probably about 20% in it will start hitting all those zero blocks, and zoom forward to completion (and since you zeroed the empty space, this will happen even with compaction turned off).

Oh yes, one final note. For you compaction is pointless anyway, because you zeroed the unused blocks as a preprocessing step, meaning that final size with and without compaction should be about the same. The point of the compact option is to save you time by not having to run sdelete or eraser first.

[Etepetete]

Thanks mpack, I really have other things to get to so I'm thankful that you relieved me of this huge test. But your tool is still converting, 7 Hr 38 min later. Still have another 55 min 20 sec to go before I test if the VDI works. From your previous post, you didn't expect this did you?

[SoftExpert]

Hi mpack, first of all thank you for this unbelievably useful tool !!!
I want to signal an issue I had when cloning and resizing a VDI:
a. Initial situation: I have a VDI of 4Gb that is almost full (92%) so I wanted to extend it to 8 Gb
b. Steps:
1. Using CloneVDI I chose to keep the GUID, to extend size to 8Gb and to compact drive
2. Launched the process
3. When successfully finished, I mounted in the VM the new VDI (replacing then old one) and mounted the gparted-live-0.4.7-1.iso
4. Started the VM, booting on gparted, and launched gparted tool
5. With gparted I resized the partition to fill up to 8Gb the disk space
6. Everything went fine, the partition is marked as having 8Gb now
7. Restarted the VM, booting this time from the C drive
8. In the Disk Management console, the partition is shown as having 8 Gb but the partition properties dialog shows 4 Gb

Before saying that is gparted's fault, creating a new VDI and copying/resizing the partition with gparted gives the expected result, the partition has its 8 Gb.

So, could it be that when resizing the VDI, something is not written as it should?
Thank you for taking so much time for helping us, the others, with such a jewel!

Best regards,
SoftExpert

[mpack]

Thanks mpack, I really have other things to get to so I'm thankful that you relieved me of this huge test.

You're welcome, and I very much appreciate you taking the time you already have.

But your tool is still converting, 7 Hr 38 min later. Still have another 55 min 20 sec to go before I test if the VDI works. From your previous post, you didn't expect this did you?

I expected a big overestimate at the beginning, however the time in your earlier post was already four hours in, by which time it may well have reached a more realistic throughput estimate.

[mpack]

Before saying that is gparted's fault, creating a new VDI and copying/resizing the partition with gparted gives the expected result, the partition has its 8 Gb.

Sorry, despite your admonition I have to insist that it's gparted's fault, or something odd about your OS.

CloneVDI does not touch the partition information, and while it's certainly possible to screw up the VDI format, no possible error I can think of would result in what you have reported. To explain: ultimately all that happens is that a field in the VDI header reports the drive size in MB, and that is the number that the guest OS or the gparted tool sees. Yes that number may be right or wrong (I claim that it's right), but if it was wrong you would get data corruption when the guest tried to use affected areas of the disk, you would not get minor misreporting of drive statistics.

[mpack]

An additional thought on this :-

At the moment I am converting from raw using your tool. Uptime 4 Hr 5 min, time remaining 4 Hr 8 min.

Given a 150GB source file, and calling it 8 hours exactly, I calculate you are getting a throughput of about 5.3MB/s - which is pretty awful! When reading and writing to the same internal drive, which is by no means very fast, I typically expect around 24MB/s sustained throughput. That USB port must really be hurting the process.

At 24MB/s your 150GB file would have been cloned in 1hr 45mins.

[Perryg]

Don,
I get between 24 MB/sec and 26MB/sec on a local drive as well. I have tested this on several OSes and even using 64 bit using Wine since it is on a Linux Host. It starts out a lot faster but then settles down to the same 24-26. I have not tested this on a USB port though, but suspect that as being the bottle neck.

[SoftExpert]

Before saying that is gparted's fault, creating a new VDI and copying/resizing the partition with gparted gives the expected result, the partition has its 8 Gb.

Sorry, despite your admonition I have to insist that it's gparted's fault, or something odd about your OS.

CloneVDI does not touch the partition information, and while it's certainly possible to screw up the VDI format, no possible error I can think of would result in what you have reported. To explain: ultimately all that happens is that a field in the VDI header reports the drive size in MB, and that is the number that the guest OS or the gparted tool sees. Yes that number may be right or wrong (I claim that it's right), but if it was wrong you would get data corruption when the guest tried to use affected areas of the disk, you would not get minor misreporting of drive statistics.

You might be right ... but how to explain the fact that the same gparted tool can copy and resize the same source partition on an new VDI without creating the issue?
Something has to be different between the two methods ... could it be that the partition table is modified in such a way that gparted does not see all the records?

Edit: I can replicate the behavior consistently; I created a 256 Mb VDI, created a NTFS partition on it, cloned the VDI with resize to 1Gb, resized the partition and the result is the same (see the screenshot) Screenshot that shows partition sizes CloneVDIPartitionIssue.png (24.8 KiB) Viewed 12062 times

[Etepetete]

mpack, I have to thank you for encouraging me to test your tool! THANK YOU!

Your tool created a dynamic drive with 150 GB virtual space with a literal size of ca. 20 GB, just like the the VBoxManage convertfromraw command. Your tool took 8 hr 37 min to complete the process. It works like a charm. But that is what we expected, isn't it?

The reason why I have thanked you is the following:

When I reported my success in that other thread I mentioned receiving Blue Screens. This time it didn't happen and I have discovered why. I had one little mistake in my script to set the DMI settings. As I mentioned before, it was in the wee hours of the morning when I finally succeeded. Well, I had written the wrong DmiBIOSFirmwareMajor variable. I had written a 3 instead of a 2. This I had corrected before I tested the VDI I created with your tool. No blue screen what so ever!

I love it! And boy, do I feel good, again!

So if your're interested in comparing conversion times to the VBoxManage convertfromraw command I'll start right away.

[Etepetete]

An additional thought on this :-
Given a 150GB source file, and calling it 8 hours exactly, I calculate you are getting a throughput of about 5.3MB/s - which is pretty awful! When reading and writing to the same internal drive, which is by no means very fast, I typically expect around 24MB/s sustained throughput. That USB port must really be hurting the process.

You are right on the mark there. Although it is an external drive.