[Invalid] modifyhd -compact NOT actually compacting .vdi file

[DThimsen]

The host system is Windows 10 pro with all updates. VirtualBox is at 5.2.16. The VM in question is a WIndows 2008 R2 server that had a problem with log creation. The 500GB dynamic .vdi file grew from approx 200GB to 420GB utilization before we corrected the problem with the log files. The file system now reports 140GB. This evening I attempted to compact the .vdi file.

I first ran sdelete before the compact:

C:\Users\Administrator>sdelete.exe c -z

SDelete - Secure Delete v1.6
Copyright (C) 1999-2010 Mark Russinovich

SDelete is set for 1 pass.
Free space cleaned on C:\
1 drives zapped

I shutdown the VM, and then ran the modifyhd command:

VBoxManage.exe modifyhd "C:\Users\admin\VirtualBox VMs\ServerNC\ServerNC.vdi" --compact

The compact ran for approximately 15 minutes and got to 100% and then returned to the command prompt with no other messages.

The host system still reports the .vdi file as 522GB. I've run the sdelete and the compact again with the same results.

Any ideas?

Thanks,
Don

[socratis]

Are you using snapshots by any chance? I'd like to see the output of the commands:

• VBoxManage showmediuminfo <YourVDi>

plus a

• dir /s

executed from the VM's directory, for example "C:\Users\<You>\VirtualBox VMs\<YourVM>".

The --compact does work, so we got to find out what's wrong with your setup.

[DThimsen]

I use mirrored drives on the VM host machine. After doing the sdelete on the VM, I issued a shutdown on the VM and watched it go to powered down status. I then powered off the host machine and removed one of the mirrorred drives and brought it home with me. The following information is from the mirror I removed before running the -compact.

C:\Program Files\Oracle\VirtualBox>VBoxManage.exe showmediuminfo "F:\Users\admin\VirtualBox VMs\ServerNC\servernc.vdi"
UUID:           7b6a8a71-3638-442e-a88f-83b5146a5010
Parent UUID:    base
State:          created
Type:           normal (base)
Location:       F:\Users\admin\VirtualBox VMs\ServerNC\servernc.vdi
Storage format: VDI
Format variant: dynamic default
Capacity:       512000 MBytes
Size on disk:   510251 MBytes
Encryption:     disabled

Code: Select all

Volume in drive F has no label.
 Volume Serial Number is 9832-DA61

 Directory of F:\Users\admin\VirtualBox VMs\ServerNC

09/20/2018  08:06 PM    <DIR>          .
09/20/2018  08:06 PM    <DIR>          ..
09/20/2018  01:16 AM    <DIR>          Logs
09/20/2018  08:06 PM             8,389 ServerNC.vbox
09/20/2018  08:05 PM             8,361 ServerNC.vbox-prev
09/20/2018  08:05 PM   535,037,136,896 ServerNC.vdi
09/20/2018  01:16 AM    <DIR>          Snapshots
               3 File(s) 535,037,153,646 bytes

 Directory of F:\Users\admin\VirtualBox VMs\ServerNC\Logs

09/20/2018  01:16 AM    <DIR>          .
09/20/2018  01:16 AM    <DIR>          ..
09/20/2018  08:05 PM           215,928 VBox.log
09/20/2018  01:15 AM           138,533 VBox.log.1
09/19/2018  08:56 PM           143,295 VBox.log.2
09/19/2018  01:15 AM           125,350 VBox.log.3
09/20/2018  08:05 PM           265,794 VBoxHardening.log
12/12/2012  01:01 PM     2,713,517,806 x2012-12-12T17-01-42-113201000Z.sav
               6 File(s)  2,714,406,706 bytes

 Directory of F:\Users\admin\VirtualBox VMs\ServerNC\Snapshots

09/20/2018  01:16 AM    <DIR>          .
09/20/2018  01:16 AM    <DIR>          ..
01/28/2018  10:09 AM     2,568,477,389 2018-01-28T14-08-54-939206800Z.sav
04/21/2018  06:05 PM     2,609,818,025 2018-04-21T22-04-55-342730300Z.sav
08/28/2018  03:15 AM     1,676,477,268 2018-08-28T07-15-01-439156100Z.sav
09/12/2018  07:05 PM     2,378,976,158 2018-09-12T23-05-53-888015000Z.sav
               4 File(s)  9,233,748,840 bytes

     Total Files Listed:
              13 File(s) 546,985,309,192 bytes
               8 Dir(s)  391,722,225,664 bytes free

[DThimsen]

Hmmm.... I just used an editor called HxD which I've used in the past to examine very large files on disk. I took a scan through the .vdi file and I'm not seeing what I expected - which should be a BUNCH of binary zeros since I ran sdelete just before breaking the mirror. Maybe I don't understand the format of a vdi file. I've never actually looked at a dynamic .vdi file before.

Should I look at using a newer version of sdelete??? I remember a few years ago when a newer version sdelete, and it had horrendous performance problems. So I went back to a version 1.6.

[socratis]

The following information is from the mirror I removed before running the -compact

OK, so that shows the original state of the VDI, which is 500 GB according to VBoxManage and roughly the same size in Windows. But that doesn't bode well with the fact that you said:

The 500GB dynamic .vdi file grew from approx 200GB to 420GB

Why did you report 420 GB when Windows clearly says 500 GB? Where did you get that 420 number from?

Second, I would like to see the after information as well. Because that's where you say the problem is after all, no? What you've shown me here is from the state of things before you even used the --compact option.

Should I look at using a newer version of sdelete???

NOOOOOOOOO!!!!!!

I remember a few years ago when a newer version sdelete, and it had horrendous performance problems

That was not performance problems, that is a potentially catastrophic bug! As in data loss bug! Stick to 1.6.

Are you sure you're not confusing the primary and the mirror drives? Why don't you forget the mirror for a moment. You don't want to be working on your data and their backup at the same time.

[DThimsen]

Why did you report 420 GB when Windows clearly says 500 GB? Where did you get that 420 number from?

This is from the perspective of the Windows Server 2008 R2 VM machine. The VM was created with a 500GB dynamic .vdi disk, but the file system (c: disk) used by the VM actually uses about 150GB. The original problem with the log files grew the file system and associated .vdi to around 420. Running SDELETE then pushed the .vdi file up to the max 500GB.

Are you sure you're not confusing the primary and the mirror drives? Why don't you forget the mirror for a moment. You don't want to be working on your data and their backup at the same time.

I split the mirror last night and brought a backup home last night. The VMHOST machine had another backup drive inserted and the mirror was rebuilt about the time I got home.

Here's the information about the .vdi file from our running vm host system:

c:\vms>VBoxManage.exe showmediuminfo "c:\Users\admin\VirtualBox VMs\ServerNC\servernc.vdi"
UUID:           7b6a8a71-3638-442e-a88f-83b5146a5010
Parent UUID:    base
State:          created
Type:           normal (base)
Location:       C:\Users\admin\VirtualBox VMs\ServerNC\ServerNC.vdi
Storage format: VDI
Format variant: dynamic default
Capacity:       512000 MBytes
Size on disk:   509990 MBytes
Encryption:     disabled
In use by VMs:  ServerNC (UUID: a36689fe-8ced-4fd4-8fbd-ac10debb06c1)

c:\Users\admin\VirtualBox VMs\ServerNC>dir /s
 Volume in drive C has no label.
 Volume Serial Number is 9832-DA61

 Directory of c:\Users\admin\VirtualBox VMs\ServerNC

09/21/2018  01:17 AM    <DIR>          .
09/21/2018  01:17 AM    <DIR>          ..
09/21/2018  01:16 AM    <DIR>          Logs
09/21/2018  01:17 AM             9,003 ServerNC.vbox
09/21/2018  01:15 AM             9,060 ServerNC.vbox-prev
09/21/2018  01:17 AM   534,763,458,560 ServerNC.vdi
09/21/2018  01:17 AM    <DIR>          Snapshots
               3 File(s) 534,763,476,623 bytes

 Directory of c:\Users\admin\VirtualBox VMs\ServerNC\Logs

09/21/2018  01:16 AM    <DIR>          .
09/21/2018  01:16 AM    <DIR>          ..
09/21/2018  05:32 AM            74,662 VBox.log
09/21/2018  01:15 AM           129,808 VBox.log.1
09/20/2018  08:41 PM           131,264 VBox.log.2
09/20/2018  08:05 PM           215,928 VBox.log.3
09/21/2018  01:16 AM           260,989 VBoxHardening.log
12/12/2012  01:01 PM     2,713,517,806 x2012-12-12T17-01-42-113201000Z.sav
               6 File(s)  2,714,330,457 bytes

 Directory of c:\Users\admin\VirtualBox VMs\ServerNC\Snapshots

09/21/2018  01:17 AM    <DIR>          .
09/21/2018  01:17 AM    <DIR>          ..
               0 File(s)              0 bytes

     Total Files Listed:
               9 File(s) 537,477,807,080 bytes
               8 Dir(s)  375,254,278,144 bytes free

Lastly, do you know much about the structure a dynamic .vdi file? After running sdelete on the VM system, and knowing that the disk should be 2/3 empty space, would you expect to see lots of binary zeros in the vdi file? Because I'm not seeing near enough binary zeros with the HxD editor.

Are you using snapshots by any chance?

This was one of your original questions. Can I ask why it matters in this case?? I had powered off the VM by issuing a Shutdown from the VM client system. At the time, as you might have noticed from the original dir /s from my mirror drive, there were true snapshots from various points of time. Technically, there wasn't a "saved state" for the VM that would have been used for a restart. I can see where a saved state VM would concern the compact request, but not a generic snapshot.

Thanks,
Don

[mpack]

Are you using snapshots by any chance?

This was one of your original questions. Can I ask why it matters in this case?

Because compaction is useless if snapshots are used, and the data you want to zero is not in the current state.

A snapshot chain is a series of difference images. You are zeroing the current state and then looking for zeros in the base state. That is a fundamental misunderstanding of how snapshots work.


Basically, if you want to manage disk space in this way then you shouldn't be using snapshots. Get rid of them and then compaction will work as you expect. Even better if you used CloneVDI instead of that Jurassic era sdelete technique.

[DThimsen]

Thanks mpack. I'm not sure where the snapshots came from, but I'm not the only person working with the VMs. I' removed the snapshots and played around a bit more, but didn't make any progress with compaction. I haven't looked at CloneVDI yet, but seem to have solved the problem by using Macrium Reflect to clone the vdi file to a new disk. The application understands the NTFS file system well enough to clone the in-use sectors. I just finished a test run and it took about an hour to create a 160GB vdi file from the original 522GB. The system booted and seems to have no data loss, and I've run a chkdsk and validated the resulting file system.

Looks like I'll get to do this on our production VM this weekend.

Thanks,
Don

[socratis]

The application understands the NTFS file system well enough to clone the in-use sectors.

Seems like you're almost describing mpack's CloneVDI, minus a bunch of other useful features...

[DThimsen]

I'll definitely check out the CloneVDI app soon. Thanks!

[DdB]

Hi,

I am not affiliated with virtualbox, but using it for years, thus ran into many of the "usual problems". My understanding:
there is a raw format for virtual disks (wasting alot of space) and there are compacted formats (like vdi). As my underlying storage uses compression anyways, i can see, by how much the vdi files can compress once they get changes from the guest (or by integrating snapshots - for that matter). What i noticed: For overall space usage, compacting the vdi is unnecessary, whereas zerofree (on linux) and SDelete (on Windows) really zero out (logically) unused/unassigned space, which can then be compressed nicely, leading to less usage on the underlying compressed filesystem. Compacting instead can (to a lesser extent though) really compact the vdi thereby making the reported filesize as seen by the host smaller. the operation has no effect whatsoever to the compressed used space of the vdi after it gets compacted AND compressed.

Some years ago, i did use Clonevdi, although the reason was more to deal with proper partitioning while copying content. Nowadays, it does not make sense for me, since the folder, where my VM's reside (and its subfolders) gets compressed by the OS (like: compression on NTFS or compression=on on ZFS/linux). BTW: The compression ratio of the VM's is at around 1.7. But i recommend not worrying about space too much. Buy another (real) disk instead and use the space from it. That is much less hassle than trying to exploit every bit to the utmost-

[socratis]

My understanding:

I'm sorry to say it, but your understanding is completely wrong. Mainly because you seem to be confusing two words: compacting and compressing. They are not the same thing/concept/operation. VDIs (and VMDKs and VHDs) are not compressed or compressible. It doesn't make sense. It would be a performance killer!

They are compactable on the other hand, and what that means is that you can 1) zero-out previously used clusters from within the guest OS (that's what 'zerofree' and 'sdelete' do), and then 2) look at the zeroed-out "clusters" of the VDI and remove them from the VDI-file on the host (that's what the --compact does). And that compacts the contents of the VDI. Without using any sort of compression. See a previous post of mine to understand how compaction works at a basic level.

there is a raw format for virtual disks (wasting alot of space)

Again a misunderstanding. Raw disk access is not creating a virtual disk, it accesses your actual, physical hard drive. There's absolutely. No. Space. Wasted. AT ALL!!!

Please read the manual if your understanding is not concrete enough, it contains a lot of information that can help you understand better what's actually going on. Start with ch. 5 Virtual storage. Also worth reading (really enlightening) is the long-written All about VDIs.

[mpack]

there is a raw format for virtual disks (wasting alot of space)

Again a misunderstanding. Raw disk access is not creating a virtual disk, it accesses your actual, physical hard drive.

To be fair, the phrase "raw format" (or raw image) has been used in the past to describe something which is quite distinct from "raw access".

Raw format is simply a flat sector dump of a disk (or partition). There is no header, no table of occupancy, no smarts at all.

The command VBoxManage convertfromraw is using this meaning of "raw".

[socratis]

But you can't access the raw format as a virtual disk from VirtualBox...

[mpack]

Not without preparation anyway. You'd have to create a VMDK descriptor for it. I have a (very) vague recollection that you can use "VBoxManage internalcommands createrawvmdk" on files as well as raw disks but I won't swear to it: I just take my own template VMDK text file and modify it.