CloneVDI tool - Discussion & Support

Discussions related to using VirtualBox on Windows hosts.
caspertone
Posts: 54
Joined: 2. Apr 2014, 10:27

Re: CloneVDI tool - Discussion & Support

Post by caspertone »

mpack wrote:I find it somewhat ironic that, AFAIK, it was me who established the convention of describing the capacity of a hard drive as capacity (IME it was previously referred to as the "logical size"), and reserving "size" to refer to the current number of bytes in the host file. It was the act of developing CloneVDI that highlighted the need for clearer terminology: i.e. because prior to CloneVDI no tool even existed (*) to change the capacity of a VDI, which meant it wasn't exactly a big topic of conversation.

(*) The only related function provided by VBoxManage in 2009 was cloning, and it was command line only. There was no option to resize a drive. And even now the VBoxManage function is called "modifymedium --resize", not "modifymedium --newcapacity", so the "capacity" terminology is not universally acknowledged as superior, though obviously I do think so.

(**) Also, I refuse to pander to the GiB crowd. 1KB in a computing context means 1024 bytes, it has always meant 1024 bytes, there has never been any confusion except among people for whom confusion is a daily fact of life. Not helped by wilful disinformation such as drive mfrs quoting disk capacities in base 10 (computers don't have fingers, 10 has no special significance), because it makes it look like a larger capacity. I personally don't see how inserting an 'i' in the middle of GB will make it eternally immune from user ignorance. 1KB=1024. 1MB=1024^2. 1GB=1024^3. And so on.
Well, I did not know it was you but I find brilliant that you started that way forward. My comment was just to encourage systematic use of capacity and size in the GUI of CloneVDI and in this thread... Later we can lobby VB developpers to deprecate --resize in favour of --newcapacity ;-)

I understand your refuse to follow the GiB crowd, but sometimes it becomes crazy to try to look to bytes/KB/MB/GB and not getting lost. I have another issue on vdi size that will ask later in another question... that is making me crazy.
caspertone
Posts: 54
Joined: 2. Apr 2014, 10:27

Re: CloneVDI tool - Discussion & Support

Post by caspertone »

caspertone wrote:Thanks @mpack for this fantastic tool I love more and more every day, and more ever as I need to use its more new/advanced features.

- Petition is to enhance a bit this part, that is, capacity/size reduction functions:
a) could you show which is the minimum capacity that will embody identified partitions?
b) do/could you plan to support check of not chopping off secondary header of GPT (https://en.wikipedia.org/wiki/GUID_Partition_Table )?
c) is current version, when reducing, "moving" GPT secondary header?
CT
Any hope with the petition?
Thanks,
CT
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: CloneVDI tool - Discussion & Support

Post by scottgus1 »

I would go so far as to say that GPT vs MBR isn't in the Virtualbox scope, just like what file system gets put on a disk is not in the scope of Seagate or Western Digital or Kingston or Mushkin. Virtualbox just provides "disks". it's up to the user & OS to decide what gets put on the disk.

Approach this problem as if the guest were a real PC with real disks. One thing to consider is, is it possible to make an existing Windows 10 booting from EFI & GPT able to boot from a MBR instead?

Finally, if you have to go through fixed, go through fixed. Set Virtualbox to make a fixed drive of the required size before you go to bed, and continue the project in the morning.

CloneVDI doesn't make fixed disks, so to clone the guest disk to fixed you'll have to use vboxmanage. Or make an empty fixed disk, attach it to the guest, then clone it with 3rd-party software inside the guest OS.
caspertone
Posts: 54
Joined: 2. Apr 2014, 10:27

Re: CloneVDI tool - Discussion & Support

Post by caspertone »

Hi again,
I have an issue whose answer I have searched north and south so I did not find an answer to my question. It is related to VDI format, and certainly I am sure Mpack is an expert on this. While is partially offtopic, it is related to CloneVDI use.

Point is...

Have a second disk in a guest. Vdi format, dynamic, type writethrough (do not wish snapshots of that vdi).
When seen in the guest, HD used storage is 46.994.952KB (so, ca 44.8GB). VDI file size is 52.205.544 (so, ca 49,8GB). All normal by now, vdi size is bigger than content due to deletions, changes, etc.
Then I use CloneVDI with the VDI to compact it. It becomes 43.088.896KB (so, ca 41.1 GB).
So, compacted vdi is 41.1GB while inside information amounts to 44.8GB (mostly video, mp4, not really compressible).

Questions
a) is VDI format conveying compressing its contents?
b) if yes, is CloneVDI compressing contents?

in some previous vdi (that I spoiled with clonevdi as I applied to the vdi and it was being snapshoted and I deleted the snapshot) I found files that windows reported a size but were really empty (zipping then produced cero size zip files). But this is not the case with the current vdi...

I am quite puzzled... any insight is really welcome!

TIA/CT
caspertone
Posts: 54
Joined: 2. Apr 2014, 10:27

Re: CloneVDI tool - Discussion & Support

Post by caspertone »

scottgus1 wrote:I would go so far as to say tha....e guest OS.
Thanks for the information and suggestions. I will try the advice.
CT

Later edit

I think I will try the idea of creating an empty vdi fixed size, attach to the VM and use A c ro nis in the guest to clone from first (dynamic) to second (fixed).
Then I can replace disk1 by disk2 in the VM and check if it boots ok.
Will try over the WE, until then I can not free the machine.
Then I will try GPT to MBR...
CT
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

caspertone wrote: Questions
a) is VDI format conveying compressing its contents?
b) if yes, is CloneVDI compressing contents?
No compression is involved. Most likely there are just some zero blocks in there. VirtualBox (and CloneVDI) doesn't allocate 1MB blocks which are filled with zeros, it simply marks the block as all zeros in the blockmap. You could consider this a form of run length encoding I guess (a simple lossless compression).

I wish you would forget all that fixed size VDI nonsense. You have already been told it will not help at all, and that in fact it is completely irrelevant. There is no functional difference between fixed and dynamic VDI, there is no test a guest could do to detect the difference so how could it possibly have an affect there? The only difference affects the host only, being WHEN the 1MB host file blocks are allocated (fixed:all blocks allocated up front. dynamic:each 1MB block is allocated when the guest first writes to that 1MB section of the disk - action transparent to the guest, not the host). So the only time you would ever use fixed is if you were setting up say an autonomous server and wanted to guarantee that it could never fail in service due to running out of host disk space. That is literally the only valid reason to ever use fixed size disks.
caspertone
Posts: 54
Joined: 2. Apr 2014, 10:27

Re: CloneVDI tool - Discussion & Support

Post by caspertone »

Thanks mpack for your patience with me.

- fixed/dynamic: I understand what you say, point is that the tool that I could use in the host to convert a fileA.vdi from GPT to MBR without loosing the windows boot inside the file.vdi claims that is unable to handle dynamic vdi´s. So I understand that from your explanations of fixed/dynamic is to use the tool inside the guest where the tool should not be affected. Probably the best approach would be to create a windows guest that has myfileA.vdi as a second drive so the tool does not need to perform its duty on a running system and then would not need to reboot the VM.

- My other problem - fully unrelated to fixed/dynamic

I am not sure I understand your answer "Most likely there are just some zero blocks in there. VirtualBox (and CloneVDI) doesn't allocate 1MB blocks which are filled with zeros, it simply marks the block as all zeros in the blockmap. You could consider this a form of run length encoding I guess (a simple lossless compression)"

Can this explain vdi with filesize at host of 41.1GB keeps inside 44.8GB? If yes, how I can check in the guest for zero blocks?

Just in case my previous explanation was unclear, I try to re-elaborate:

My problem here is that I have a vdi, let me call it fileC.vdi, that, when mounted as second disk in a VM, windows reports used space as 44.8GB and the host sees fileC.vdi with a filesize of 41.1GB - as said, 99% of files inside fileC.vdi are mp4 that can not be compressed really, those files have been checked using ffmpeg and are ok.

fileC.vdi comes from compacting fileB.vdi with CloneVdi 4.01; fileB.vdi filesize is 49.8GB, so compresion produces a 41.1GB vdi.

How is it possible that fileC.vdi with filesize 41.1GB keeps inside 44.8GB? that is making me crazy....

[only answer, if compression is not involved, is that filec.vdi has inside 3.7GB of garbage that cheats windows at the guest level (but not to clonevdi or the host), something that could be a corrupted MFT, ghost files (empty but reporting a size, I already saw those with damaged mp4), I do not know... but that gets me nervous...]
My current approach now is to mount the .vdi at guest level and take a copy of the information to another hard disk...

Thanks again!
caspertone
Posts: 54
Joined: 2. Apr 2014, 10:27

Re: CloneVDI tool - Discussion & Support

Post by caspertone »

Well, I think I found the explanation, of course it is related to the zero blocks topic pointed by mpack.

Just in case anyone else touches this issue, the record of my views:

In the drive there are some P2P leftover files, temporary files. While windows reports that folder at 5.44GB, a zip compression goes to 1.6GB. The difference is 3.84GB, quite close to the disonance in vdi size and window reporting size, 3,71GB (of course 7zip will use a different compresssion approach that that of vdi ("VirtualBox (and CloneVDI) doesn't allocate 1MB blocks which are filled with zeros, it simply marks the block as all zeros in the blockmap. You could consider this a form of run length encoding").

Just to ensure findings, I also:

- I mounted both vdi (before/after compression) (arsenal image mounter has a free version that allows to mount and show vdi as letter units in windows), and did a byte by byte comparison of files (freefilesync, also free). 100% equal files.

- searched about sparse files in windows. but I had not marked files as sparse...

- found that "The files with the extension “.part” are downloads in P2P sw that are not yet finished... part files always have the
size of the final download. The missing parts are zerofilled. In the newer versions, and when using the NTFS file system, you have the option to share your incomplete downloads as “sparse”. This counteracts the mentioned process and therefore saves space on your hard disk."

All in all, it looks that the P2P sw creates and uses files "somewhat as sparse" but on its own, without operating system support. That will explain that file size and file on disk size reported by windows are the same. And, as indicated by mpack, the compaction of the vdi removes that zerofiled areas of the .part files.

Looks that it is to be expected that compacted vdis with .part files will underperform with P2P sw as they will not get advantage of their zerofill strategy.

Last check has been to remove the P2P files in the non compacted vdi, compact, and check sizes... Yes, now compacted vdi file size (host) is not smaller that windows reported contents size (guest).

CONCLUSIONS
Now I fully understand mpack statement "doesn't allocate 1MB blocks which are filled with zeros, it simply marks the block as all zeros in the blockmap", and I found the culprit/origin of the unexplained non-fitting size... .part files are "somewhat sparse" at application level. Now I will sleep better...

Thanks all.

Later edit:
.part files / "somewhat sparse" at application level - I mean
a) that the OS is not aware of the file being sparse. NTFS manages sparse files, taking the advantage of not consuming disk space. But, files have to be marked as sparse, otherwise the OS is not aware that size on disk is smaller that file size. In the case of .part files, the P2P program, unless configured as such, does not mark .part files as sparse to the OS
b) as far as I understand, .part files are not true sparse files (https://en.wikipedia.org/wiki/Sparse_file), as the program allocates the file with full size (so no advantage of non consuming disk space) but the program uses sparse files techniques probably to be fast and to avoid disk full situations, and perhaps to fill in incoming new information. Sparseness/hole punching have their complexities I will not dig in... whoever interested can look here https://stackoverflow.com/questions/139 ... it-be-used and here https://stackoverflow.com/questions/385 ... zero-range ; at the end of the day, vdi related, is that CloneVDI identifies zerofilled/nonallocated parts inside the vdi (both at the not file-assigned areas AND inside files) and makes it magic of compacting - CAVEAT EMPTOR: I simply do not know if the compaction has functional impact on the "somewhat sparse" .part files... but looks to me that the P2P program I use is able to go on...

Should you wish to check (Windows) if a file is "somewhat sparse", there is a program called sparse_checker, that can even convert the file in "OS sparse". It is 32bit and you would need VB2015dlls, and to retrieve it from wayback timemachine - here - https://web.archive.org/web/20110602032 ... ecker.html Note that NTFS has to be used (spare aware FS). Did a test with a 873KB size and size on disk .part file and it went to 813 size and 513KB size on disk.

OVER, promise
wmeyer
Posts: 66
Joined: 14. May 2009, 18:42
Primary OS: MS Windows 7
VBox Version: OSE other
Guest OSses: Windows 7

Re: CloneVDI tool - Discussion & Support

Post by wmeyer »

I have tried on a few occasions to create a new Windows 10 VM in VMware and then to use CloneVDI to clone a VDI from it. Although the VMware machine worked fine, the VBox VDI conversion reports boot failure. I wonder whether this is a known issue, and/or whether there is some workaround or repair?

Thanks,

Bill
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

No, it's not a known issue, and you haven't told me anything yet that makes me think it's a cloning issue. You haven't even said what options were used during cloning. Do you have some idea what's wrong with the disks?
wmeyer
Posts: 66
Joined: 14. May 2009, 18:42
Primary OS: MS Windows 7
VBox Version: OSE other
Guest OSses: Windows 7

Re: CloneVDI tool - Discussion & Support

Post by wmeyer »

Options used:
Generate new UUID

Error on running is:
FATAL: Int 18: BOOT FAILURE

I have attached the log. The VMDK works fine in VMware, so to the extent of my knowledge, there is no issue with that.
Both the VMDK and the converted VDI are running on an SSD, which I would guess is not a factor.
I set minimal options in VBox: 2GB of RAM, 2 processors.

If there is anything else I can do here to isolate the issue, I will be happy to do so.
Attachments
VBox.7z
log of attempted boot
(20.05 KiB) Downloaded 62 times
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

"Generate new UUID" was precisely the setting I was wondering about. GPT partitions have lots more security features than MBR used to have, and Win10 has a more complicated boot including identifying boot partition by UUID, so I frankly don't know if it's ok to move a VMWare GPT disk to a new VirtualBox disk controller and change the UUID.

Acid test 1: use the VMDK directly. Note however that this preserves the UUID, so it's not completely comparable.

Acid test 2: would be to clone a VirtualBox GPT disk and use it in VirtualBox, I'm trying that now (copying will take another couple of minutes). If I don't get the problem then cloning is not the issue, it's probably the platform / UUID change.

Acid test 3: would be to clone the disk using VBoxManage. If you still get the problem when CloneVDI is not involved then it obviously isn't a CloneVDI problem.

Acid test 4: variant of test two, but don't change the UUID. I expect this to work. We'll see soon.
 Edit:  I just completed test 2 and the cloned disk (containing Win10-64bit of course) worked perfectly, even though the UUID changed. This means there's no need for me to do tests 3 and 4.

Perhaps some kind of fixboot tool would work, since I'm confident that all you have is a partition/controller ID issue and not some fundamental problem with the clone file format. 
martin123
Posts: 61
Joined: 21. Nov 2020, 11:30

Re: CloneVDI tool - Discussion & Support

Post by martin123 »

==
P2V feature
==

The app's doc mentions that we can use P2V feature, as follow:
Using CloneVDI to do a P2V ("Physical To Virtual" Disk Conversion)
CloneVDI now recognizes when you have used a Windows disk device name as the source filename (for
example, "\\.\PhysicalDrive0" identifies the boot drive on most Windows hosts), and slightly modifies
its behaviour in that case to make it possible to clone the physical host hard disk to a VDI file. ....... On some Windows hosts you may also need to
run CloneVDI with elevated privileges ("Run as administrator") since this feature requires CloneVDI to
have sector level (read only) access to the source drive.
I runned the app as administrator, I used the syntax as mentionned in the exemple above and many other, but I can't select the C:\ driver.
Which syntax do we have to execute exactly?

Thank You.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

Can you tell me exactly what you typed? And what error message you got?

C: is not a drive on modern Windows, it's a label attached to one partition. You can't use that to identify a drive. A Windows drive ID will have a syntax similar to the example given in the release notes, though you should check how drives are actually numbered on your PC.

Also, this is not a beginner-friendly feature in CloneVDI. Disk2VHD might suit you better for this, and you can still use CloneVDI to convert the VHD to VDI before use.
martin123
Posts: 61
Joined: 21. Nov 2020, 11:30

Re: CloneVDI tool - Discussion & Support

Post by martin123 »

Sorry, It was a newbie question. I finally succeeded to create a VDI hard disk drive clone.
I used the formula as you mentionned and chose a folder destination to proceed cloning as shown in pic bellow.

Then, I created a virtual machine on VirtualBox. But when I started that VM, I had neither the desktop icons nor the Tasks Bar. The desktop was empty.!

== ==
I already used Disk2VHD. The VHD was saved on external hard drive because of its 75 GB size. The issue was when implemented it in VirtualBox, after logout & login from virtual machine, the VHD file disappeared from the hard drive! It was wiped out itself! I can't found it.

== ==

== ==
Attachments
pic.jpg
pic.jpg (57.46 KiB) Viewed 41175 times
Post Reply