CloneVDI tool - Discussion & Support

Discussions related to using VirtualBox on Windows hosts.

Re: CloneVDI tool - Discussion & Support

Postby caspertone » 22. Sep 2020, 20:32

mpack wrote:I find it somewhat ironic that, AFAIK, it was me who established the convention of describing the capacity of a hard drive as capacity (IME it was previously referred to as the "logical size"), and reserving "size" to refer to the current number of bytes in the host file. It was the act of developing CloneVDI that highlighted the need for clearer terminology: i.e. because prior to CloneVDI no tool even existed (*) to change the capacity of a VDI, which meant it wasn't exactly a big topic of conversation.

(*) The only related function provided by VBoxManage in 2009 was cloning, and it was command line only. There was no option to resize a drive. And even now the VBoxManage function is called "modifymedium --resize", not "modifymedium --newcapacity", so the "capacity" terminology is not universally acknowledged as superior, though obviously I do think so.

(**) Also, I refuse to pander to the GiB crowd. 1KB in a computing context means 1024 bytes, it has always meant 1024 bytes, there has never been any confusion except among people for whom confusion is a daily fact of life. Not helped by wilful disinformation such as drive mfrs quoting disk capacities in base 10 (computers don't have fingers, 10 has no special significance), because it makes it look like a larger capacity. I personally don't see how inserting an 'i' in the middle of GB will make it eternally immune from user ignorance. 1KB=1024. 1MB=1024^2. 1GB=1024^3. And so on.


Well, I did not know it was you but I find brilliant that you started that way forward. My comment was just to encourage systematic use of capacity and size in the GUI of CloneVDI and in this thread... Later we can lobby VB developpers to deprecate --resize in favour of --newcapacity ;-)

I understand your refuse to follow the GiB crowd, but sometimes it becomes crazy to try to look to bytes/KB/MB/GB and not getting lost. I have another issue on vdi size that will ask later in another question... that is making me crazy.
caspertone
 
Posts: 54
Joined: 2. Apr 2014, 10:27

Re: CloneVDI tool - Discussion & Support

Postby caspertone » 22. Sep 2020, 20:35

caspertone wrote:Thanks @mpack for this fantastic tool I love more and more every day, and more ever as I need to use its more new/advanced features.

- Petition is to enhance a bit this part, that is, capacity/size reduction functions:
a) could you show which is the minimum capacity that will embody identified partitions?
b) do/could you plan to support check of not chopping off secondary header of GPT (https://en.wikipedia.org/wiki/GUID_Partition_Table )?
c) is current version, when reducing, "moving" GPT secondary header?
CT


Any hope with the petition?
Thanks,
CT
caspertone
 
Posts: 54
Joined: 2. Apr 2014, 10:27

Re: CloneVDI tool - Discussion & Support

Postby scottgus1 » 22. Sep 2020, 20:40

I would go so far as to say that GPT vs MBR isn't in the Virtualbox scope, just like what file system gets put on a disk is not in the scope of Seagate or Western Digital or Kingston or Mushkin. Virtualbox just provides "disks". it's up to the user & OS to decide what gets put on the disk.

Approach this problem as if the guest were a real PC with real disks. One thing to consider is, is it possible to make an existing Windows 10 booting from EFI & GPT able to boot from a MBR instead?

Finally, if you have to go through fixed, go through fixed. Set Virtualbox to make a fixed drive of the required size before you go to bed, and continue the project in the morning.

CloneVDI doesn't make fixed disks, so to clone the guest disk to fixed you'll have to use vboxmanage. Or make an empty fixed disk, attach it to the guest, then clone it with 3rd-party software inside the guest OS.
scottgus1
Site Moderator
 
Posts: 9420
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: CloneVDI tool - Discussion & Support

Postby caspertone » 22. Sep 2020, 20:49

Hi again,
I have an issue whose answer I have searched north and south so I did not find an answer to my question. It is related to VDI format, and certainly I am sure Mpack is an expert on this. While is partially offtopic, it is related to CloneVDI use.

Point is...

Have a second disk in a guest. Vdi format, dynamic, type writethrough (do not wish snapshots of that vdi).
When seen in the guest, HD used storage is 46.994.952KB (so, ca 44.8GB). VDI file size is 52.205.544 (so, ca 49,8GB). All normal by now, vdi size is bigger than content due to deletions, changes, etc.
Then I use CloneVDI with the VDI to compact it. It becomes 43.088.896KB (so, ca 41.1 GB).
So, compacted vdi is 41.1GB while inside information amounts to 44.8GB (mostly video, mp4, not really compressible).

Questions
a) is VDI format conveying compressing its contents?
b) if yes, is CloneVDI compressing contents?

in some previous vdi (that I spoiled with clonevdi as I applied to the vdi and it was being snapshoted and I deleted the snapshot) I found files that windows reported a size but were really empty (zipping then produced cero size zip files). But this is not the case with the current vdi...

I am quite puzzled... any insight is really welcome!

TIA/CT
caspertone
 
Posts: 54
Joined: 2. Apr 2014, 10:27

Re: CloneVDI tool - Discussion & Support

Postby caspertone » 22. Sep 2020, 20:52

scottgus1 wrote:I would go so far as to say tha....e guest OS.

Thanks for the information and suggestions. I will try the advice.
CT

Later edit

I think I will try the idea of creating an empty vdi fixed size, attach to the VM and use A c ro nis in the guest to clone from first (dynamic) to second (fixed).
Then I can replace disk1 by disk2 in the VM and check if it boots ok.
Will try over the WE, until then I can not free the machine.
Then I will try GPT to MBR...
CT
caspertone
 
Posts: 54
Joined: 2. Apr 2014, 10:27

Re: CloneVDI tool - Discussion & Support

Postby mpack » 23. Sep 2020, 09:42

caspertone wrote:Questions
a) is VDI format conveying compressing its contents?
b) if yes, is CloneVDI compressing contents?

No compression is involved. Most likely there are just some zero blocks in there. VirtualBox (and CloneVDI) doesn't allocate 1MB blocks which are filled with zeros, it simply marks the block as all zeros in the blockmap. You could consider this a form of run length encoding I guess (a simple lossless compression).

I wish you would forget all that fixed size VDI nonsense. You have already been told it will not help at all, and that in fact it is completely irrelevant. There is no functional difference between fixed and dynamic VDI, there is no test a guest could do to detect the difference so how could it possibly have an affect there? The only difference affects the host only, being WHEN the 1MB host file blocks are allocated (fixed:all blocks allocated up front. dynamic:each 1MB block is allocated when the guest first writes to that 1MB section of the disk - action transparent to the guest, not the host). So the only time you would ever use fixed is if you were setting up say an autonomous server and wanted to guarantee that it could never fail in service due to running out of host disk space. That is literally the only valid reason to ever use fixed size disks.
mpack
Site Moderator
 
Posts: 32110
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Postby caspertone » 23. Sep 2020, 10:41

Thanks mpack for your patience with me.

- fixed/dynamic: I understand what you say, point is that the tool that I could use in the host to convert a fileA.vdi from GPT to MBR without loosing the windows boot inside the file.vdi claims that is unable to handle dynamic vdi´s. So I understand that from your explanations of fixed/dynamic is to use the tool inside the guest where the tool should not be affected. Probably the best approach would be to create a windows guest that has myfileA.vdi as a second drive so the tool does not need to perform its duty on a running system and then would not need to reboot the VM.

- My other problem - fully unrelated to fixed/dynamic

I am not sure I understand your answer "Most likely there are just some zero blocks in there. VirtualBox (and CloneVDI) doesn't allocate 1MB blocks which are filled with zeros, it simply marks the block as all zeros in the blockmap. You could consider this a form of run length encoding I guess (a simple lossless compression)"

Can this explain vdi with filesize at host of 41.1GB keeps inside 44.8GB? If yes, how I can check in the guest for zero blocks?

Just in case my previous explanation was unclear, I try to re-elaborate:

My problem here is that I have a vdi, let me call it fileC.vdi, that, when mounted as second disk in a VM, windows reports used space as 44.8GB and the host sees fileC.vdi with a filesize of 41.1GB - as said, 99% of files inside fileC.vdi are mp4 that can not be compressed really, those files have been checked using ffmpeg and are ok.

fileC.vdi comes from compacting fileB.vdi with CloneVdi 4.01; fileB.vdi filesize is 49.8GB, so compresion produces a 41.1GB vdi.

How is it possible that fileC.vdi with filesize 41.1GB keeps inside 44.8GB? that is making me crazy....

[only answer, if compression is not involved, is that filec.vdi has inside 3.7GB of garbage that cheats windows at the guest level (but not to clonevdi or the host), something that could be a corrupted MFT, ghost files (empty but reporting a size, I already saw those with damaged mp4), I do not know... but that gets me nervous...]
My current approach now is to mount the .vdi at guest level and take a copy of the information to another hard disk...

Thanks again!
caspertone
 
Posts: 54
Joined: 2. Apr 2014, 10:27

Re: CloneVDI tool - Discussion & Support

Postby caspertone » 23. Sep 2020, 23:37

Well, I think I found the explanation, of course it is related to the zero blocks topic pointed by mpack.

Just in case anyone else touches this issue, the record of my views:

In the drive there are some P2P leftover files, temporary files. While windows reports that folder at 5.44GB, a zip compression goes to 1.6GB. The difference is 3.84GB, quite close to the disonance in vdi size and window reporting size, 3,71GB (of course 7zip will use a different compresssion approach that that of vdi ("VirtualBox (and CloneVDI) doesn't allocate 1MB blocks which are filled with zeros, it simply marks the block as all zeros in the blockmap. You could consider this a form of run length encoding").

Just to ensure findings, I also:

- I mounted both vdi (before/after compression) (arsenal image mounter has a free version that allows to mount and show vdi as letter units in windows), and did a byte by byte comparison of files (freefilesync, also free). 100% equal files.

- searched about sparse files in windows. but I had not marked files as sparse...

- found that "The files with the extension “.part” are downloads in P2P sw that are not yet finished... part files always have the
size of the final download. The missing parts are zerofilled. In the newer versions, and when using the NTFS file system, you have the option to share your incomplete downloads as “sparse”. This counteracts the mentioned process and therefore saves space on your hard disk."

All in all, it looks that the P2P sw creates and uses files "somewhat as sparse" but on its own, without operating system support. That will explain that file size and file on disk size reported by windows are the same. And, as indicated by mpack, the compaction of the vdi removes that zerofiled areas of the .part files.

Looks that it is to be expected that compacted vdis with .part files will underperform with P2P sw as they will not get advantage of their zerofill strategy.

Last check has been to remove the P2P files in the non compacted vdi, compact, and check sizes... Yes, now compacted vdi file size (host) is not smaller that windows reported contents size (guest).

CONCLUSIONS
Now I fully understand mpack statement "doesn't allocate 1MB blocks which are filled with zeros, it simply marks the block as all zeros in the blockmap", and I found the culprit/origin of the unexplained non-fitting size... .part files are "somewhat sparse" at application level. Now I will sleep better...

Thanks all.

Later edit:
.part files / "somewhat sparse" at application level - I mean
a) that the OS is not aware of the file being sparse. NTFS manages sparse files, taking the advantage of not consuming disk space. But, files have to be marked as sparse, otherwise the OS is not aware that size on disk is smaller that file size. In the case of .part files, the P2P program, unless configured as such, does not mark .part files as sparse to the OS
b) as far as I understand, .part files are not true sparse files (https://en.wikipedia.org/wiki/Sparse_file), as the program allocates the file with full size (so no advantage of non consuming disk space) but the program uses sparse files techniques probably to be fast and to avoid disk full situations, and perhaps to fill in incoming new information. Sparseness/hole punching have their complexities I will not dig in... whoever interested can look here https://stackoverflow.com/questions/139 ... it-be-used and here https://stackoverflow.com/questions/385 ... zero-range ; at the end of the day, vdi related, is that CloneVDI identifies zerofilled/nonallocated parts inside the vdi (both at the not file-assigned areas AND inside files) and makes it magic of compacting - CAVEAT EMPTOR: I simply do not know if the compaction has functional impact on the "somewhat sparse" .part files... but looks to me that the P2P program I use is able to go on...

Should you wish to check (Windows) if a file is "somewhat sparse", there is a program called sparse_checker, that can even convert the file in "OS sparse". It is 32bit and you would need VB2015dlls, and to retrieve it from wayback timemachine - here - https://web.archive.org/web/20110602032 ... ecker.html Note that NTFS has to be used (spare aware FS). Did a test with a 873KB size and size on disk .part file and it went to 813 size and 513KB size on disk.

OVER, promise
caspertone
 
Posts: 54
Joined: 2. Apr 2014, 10:27

Previous

Return to VirtualBox on Windows Hosts

Who is online

Users browsing this forum: No registered users and 44 guests