Dynamically allocated storage exploded to full size without warning

[emcdell]

I created a Windows 10 64-bit guest on my Windows 7 64-bit host
The guest VDI was set to 465GB Virtual size, Type (Format) Normal and Dynamically allocated storage
Shortly after the installation completed, I checked and the VDI actual size (both in the VBox GUI and the file size on disk) and it was around 22GB and holding.
At some point after that, the VDI started growing as a background task to finally reach the full actual size.

When the new Windows 10 guest started spontaneously expanding, it filled up the 256GB SSD and the guest shut down due to a full host disk.
I copied everything to a 512GB SSD and restarted the guest.
The new Windows 10 guest started expanding again and filled the 512GB SSD and the guest shut down due to a full host disk.

Now, everything is running as expected on a 1TB SSD and is stable.
Except the Win 10 guest VDI file is at 465GB.

The Win 10 guest system shows 395GB free of 465GB available (approx 70GB used).

3 other guests (2 Windows 32-bit and Linux 64-bit) on this host are configured to use a Dynamically allocated storage:
Windows 7: 128GB Virtual / 85GB Actual
Windows 7: 126GB Virtual / 125.96GB Actual
SuSE 12: 32GB Virtual / 2.3GB Actual
All three systems shared the same original SSD drive (256GB)

Over the last 5 years, I have used several different versions of VirtualBox on several different hosts (Linux, MacOS and Windows) using a mix of Windows and Linux guests and I have always selected the Dynamically allocated storage for the VDI
This is the only time I have seen a Dynamically allocated storage mysteriously expand to the full "actual" size on any VM

Questions:
Assuming that the Windows 10 Installer didn't actually load 460GB of data onto the VM (and then deleted most of it): What may have caused the to spontaneously expand?
Can I shrink the VDI back to a more appropriate actual size (there is only 70GB or so of actual data on the VDI)?

Details:
VirtualBox Version 5.1.30 r118389 (Qt5.6.2)
Windows 7 Enterprise, SP1, latest MS updates installed, 64-bit
Lenovo WS530 laptop, 2 CPU (i7) with 8 core total, 32GB RAM
C: drive 500GB Samsung SSD (internal, primary disk)
D: drive 1000GB WD Blue SSD (all VM are on this disk) (internal CDROM to HD carrier)


Thanks.

[socratis]

What may have caused the to spontaneously expand?

You have been using the VM, that's all it takes. When an OS asks for a new allocation, it usually occupies a previously unoccupied cluster. When it "deletes" data, it simply marks the cluster as empty, internally. VirtualBox will allocate an actual "cluster" on the host when asked, but it can't know when a "cluster" is freed. So keep writing/deleting data (from temp to cache) will get your dynamically sized VDI to its maximum size.

So, from time to time you got to compact your VDI. You can either use CloneVDI, or the "VBoxManage modifymedium" command with the "--compact" switch. You have to use an appropriate zero deletion tool beforehand, depending on your guest. For more information, see ch. 8.23 VBoxManage modifymedium, at the --compact option.

[emcdell]

When an OS asks for a new allocation, it usually occupies a previously unoccupied cluster. When it "deletes" data, it simply marks the cluster as empty, internally. VirtualBox will allocate an actual "cluster" on the host when asked, but it can't know when a "cluster" is freed.

Thank you for your reply. I respect your comments and I will look into some of this.

But (respectfully)...
I am pretty sure I understand the concept of thin provisioning. It has been around for quite a while and it is used in several products that I have experience with.
And, I get the concept that it is scale-out in terms of capacity based on the guest's space requirements and that it does not automatically scale-down.
It is good to know there may be a manual scale-down procedure.
I will look into the "zero deletion tool". I assume that the tool is similar the disk cleanup tool that is included in Windows... Only better.

So keep writing/deleting data (from temp to cache) will get your dynamically sized VDI to its maximum size

Are you suggesting that Windows 10 (unlike Windows 7, Windows XP or 1/2 dozen Linuxs I've run on VBox) "asks" for allocation regardless of whether it needs space or not?
At the end of the install, it was only 16GB to 18GB used.

Also, why would the expansion of the VDI increase in large chucks of disk space (approx 300MB to 500MB chucks at a time) at 3 to 5 second internals? This seems like something in VBox triggered this external to the guest OS.

You have been using the VM, that's all it takes.

Also, I have not been using this VM much at all.
I installed the OS on Thursday of last week and the explosion of the VDI commenced within hours after the install completed.
There were several hours spent with the gust shutdown so I could copy the 100GB+ VDI to the 1st new SSD.
Then another several hours copying the next 300GB+ file to the next SSD.
Then several more hours spent coping the 465GB file to the final SSD.
During this series of copying the ever growing VDI to a series of newer, larger disks, the guest was "powered on" for a total of maybe 10 hours while I watched the VDI continue to grow and hoping it would stop at some point.
So, I really was not "using it"

The 2 Windows 7 guests that are on this host literally have years of actual use.

3 other guests (2 Windows 32-bit and Linux 64-bit) on this host are configured to use a Dynamically allocated storage:
Windows 7: 128GB Virtual / 85GB Actual
Windows 7: 126GB Virtual / 125.96GB Actual
SuSE 12: 32GB Virtual / 2.3GB Actual
All three systems shared the same original SSD drive (256GB)

[emcdell]

FYI: The documentation at manual/ch08.html#vboxmanage-modifyvdi indicates using 'sdelete' for Windows

For Windows guests, you can use the sdelete tool provided by Microsoft. Execute sdelete -z in the guest to zero the free disk space before compressing the virtual disk image

The sdelete command is not included with Windows 10, but is available for download from MS:
/sysinternals/downloads/sdelete

However, it seems to have a problem on my guest.

Run the Command Prompt as "administrator"

C:\windows\system32>sdelete /?

SDelete v2.0 - Secure file delete
Copyright (C) 1999-2016 Mark Russinovich
Sysinternals - xxx

usage: sdelete [-p passes] [-r] [-s] [-q] <file or directory> [...]
sdelete [-p passes] [-z|-c [percent free]] <drive letter [...]>
sdelete [-p passes] [-z|-c] <physical disk number>
-c Clean free space. Specify an option amount of space
to leave free for use by a running system.
-p Specifies number of overwrite passes (default is 1)
-r Remove Read-Only attribute
-s Recurse subdirectories
-z Zero free space (good for virtual disk optimization)
-nobanner Do not display the startup banner and copyright message.

Disks must not have any volumes in order to be cleaned.

C:\windows\system32>sdelete -z c

SDelete v2.0 - Secure file delete
Copyright (C) 1999-2016 Mark Russinovich
Sysinternals - xxx

SDelete is set for 1 pass.

Cleaning disk C:

Error opening disk C:
The system cannot find the file specified.

C:\windows\system32>sdelete -z 0

SDelete v2.0 - Secure file delete
Copyright (C) 1999-2016 Mark Russinovich
Sysinternals - xxx

SDelete is set for 1 pass.

Cleaning disk 0:

Error opening disk 0:
The process cannot access the file because it is being used by another process.


C:\windows\system32>

So, is this some kind of "Chicken or the Egg" type problem?

[BillG]

From a Windows host, I have always found the best way to compact a virtual disk is to use mpack's CloneVDI tool with the Compact drive while copying option set. It is simple to use, fast and informative. I would expect a reasonable reduction in size even without doing sdelete first.

Details on CloneVDI are in a sticky at the top of this forum. There are several ways to do the job, depending on the UUID option you use. When you are creating a clone you usually want a new UUID, so that is the default. If you use that option when compacting, you will need remove the old .vdi file from the Virtual Media Manager before you can add the compacted one, even if you give it the same name. If you alter that option so that the new .vdi file has the same UUID as the old one, you need only rename the clone to have the old name (after you change the name of the old file to make that name available again). The good news is that the old file is still there unaltered, so you can easily get back to where you were if things go astray.

One problem in this case is that the .vdi is so large that you may need to create the cloned file on a different drive (another reason to compact your .vdi before it gets really big). I would experiment first with one of the other vms which is still a reasonable size.

[emcdell]

Assuming this weird explosion of the VDI to actual full size was an anomaly, would it be better to backup the guest OS using Windows backup, create a new VM (with a smaller virtual disk defined) and then restore to the new VM?
I think Windows backup will restore to a smaller drive.
And then, hope the VDI new does not explode.

I'm running out of "spare" disks.
I have a USB 3.0 attached 1TB HDD, but that is very slow (less then 100MB /sec) and takes hours.

[BillG]

That might work, but I can't see how it is easier or faster. Have you actually tried CloneVDI to see how easy it is? I just ran a test. A 16G .vdi took just over 2 minutes to process SSD to SSD on an i7. All that remained was to rename the files and restart the vm. Even if you had to run it to an external USB 3.0 drive and then copy the cloned .vdi back I would bet you would be done long before Windows Backup got very far on a large fragmented drive.

On the other matter, sdelete is not like the disk cleanup tool. When you remove a file, all that really happens is that the name is removed from the index. The data is still there. Sdelete zeroes out the data area so that a compacting program can release the space.

[mpack]

Assuming this weird explosion of the VDI to actual full size was an anomaly

It was no anomaly, at least not one from VirtualBox. It happened because you did something in the guest which involved writing to every cluster in the virtual drive. That is the only reason why a dynamic VDI ever grows. If you write to every cluster then the VDI expands to full size. We can't tell you what the something was, but known culprits are: background disk defrag (don't do defrag unless you follow it with a VDI compact), disk indexing (wouldn't cause explosive growth?), "thorough disk check", disk zero fill (e.g. sdelete).

[emcdell]

TIt happened because you did something in the guest which involved writing to every cluster in the virtual drive. That is the only reason why a dynamic VDI ever grows. If you write to every cluster then the VDI expands to full size

Got it.
You may not have noticed in the top of this thread - it is a new install
There was no defrag, indexing or an attempt to run sdelete
Do you have any other examples of something I may have installed that will write to every cluster on a new disk (physical or virtual)?
Any ideas?

I did download sdelete and the program ran, but threw error messages (as described above) and quit without doing anything on the guest OS disk.
While my characterization of sdelete may have been a little off, the more precise term may be "garbage collection".

The program CloneVDI did function. It ran quick enough, but in the end, the clone was exactly the same size as the original and maxed out 500GB SSD I copied it to.
And, the "Compact drive while copying" option was checked, but it did not reduce the size of the VDI file one single byte.

At this point, I will have to agree that something indeed caused the guest OS to ask for more space - but I don't have any clues what it may have been.
I will also assume that the failure of sdelete to find a "C drive" or access "drive 0" and the inability of CloneVDI to compact the VDI may be somehow related to whatever that condition is.

I guess it will remain a mystery.

I hesitate reinstalling due to licensing and hostname tracking.
If I did a reinstall, I would choose a smaller Dynamically allocated storage VDI. Just in case.

Thanks.

[socratis]

There are no mysteries here. VirtualBox does whatever the guest OS tells it to do. It doesn't do things on its own, that's a fact. I don't really know how Win10 works on the inside, do you? I'll bet on "no", so accept the facts that it may not work as other guests. I have to say though, that I've had several Win10 installations and I've never had the issue that you're describing...

Look for temp files, leftover from the installation files, etc. I am not sitting in front of your computer, you are. I can't solve it from here...

SDelete is definitely not a garbage collection, it's you that didn't use it properly. Try "sdelete c: -z" and tell me if it fails. You're going to be the first that it has failed. The only glitch is that the latest versions seem to get stuck at 100% for some reason. You can Ctrl-C it at that point with no problems.

[mpack]

The name "sdelete" is a contraction of "secure delete". The purpose is to zero fill file or deleted file areas to ensure that data sectors can't be recovered. E.g. you might do this if you were selling a PC. It has nothing to do with garbage collection. Using sdelete on a VDI precisely involves writing to every unused cluster on the drive and will make the VDI expand to full size.

As to CloneVDI compaction not working: it does indeed work if you have a supported filesystem (listed in the readme). It's possible that you enabled EFI and GPT partitioning in your Win10 guest, in which case CloneVDI does not currently list GPT as a supported filesystem, it only supports MBR partitioning. However the traditional method (the only one available using VBoxManage) of sdelete followed by cloning still works.

[socratis]

The name "sdelete" is a contraction of "secure delete". The purpose is to zero fill file or deleted file areas to ensure that data sectors can't be recovered.

There are multiple ways that sdelete works. One of them (-z) is used for VMs in order to clean the free space. More info at: https://docs.microsoft.com/en-us/sysint ... ds/sdelete

Using sdelete on a VDI precisely involves writing to every unused cluster on the drive and will make the VDI expand to full size.

No it won't, I've done that a lot of times. It will clear (zero out) only the sectors that were used, it won't overwrite every single sector. Again, that's with the "-z" switch.

[mpack]

It will clear (zero out) only the sectors that were used, it won't overwrite every single sector. Again, that's with the "-z" switch.

I don't see how that's possible. Last time I looked, sdelete (in the relevant mode) worked by creating a max sized file which it then filled with zeros. Without analyzing the filesystem at sector level (not possible AFAIK without a kernel driver or making the drive offline), it wouldn't be possible to know which sectors outside of files were previously used. It can only tell which ones are currently used.

The reason the VDI doesn't always expand to the max is because VirtualBox has code to detect that a VDI block is filled with zeros, I guess that could come into play here so I should perhaps have mentioned this as a caveat.

[socratis]

worked by creating a max sized file which it then filled with zeros.

No, I must insist on that. It would have failed on my system multiple times, time after time, so the max size creation is not happening. That, I know for sure.

it wouldn't be possible to know which sectors outside of files were previously used. It can only tell which ones are currently used.

Thinking out loud here, but, seeing which sectors are not currently used, and fill those with zeroes so that the "--compact" can actually free them? They do have a "How SDelete Works" section, but it's way above my pay grade...

[mpack]

What does "failed on your system" mean? Are you under the impression that creating a max sized guest file would by itself max out the VDI? It does not. All it involves is creating an entry in a filesystem directory and marking all the clusters as used in the cluster bitmap. The VDI on the host would grow minimally, if at all.

Blocks in a VDI are only allocated when the block is written to for the first time, not for any other reason. Creating a file directory entry does not require writing to all file blocks.