Disk Size Full Error - But Have Ample Space

[Saravanan Kandasamy]

VirtualBox 4.2.18 r88780
Host - Windows 8
Guest - Ubuntu 10.4

I have 1TB configured VDI (dynamic) on a 350GB physical hard disk. The actual space used by the guest is 36GB, obtained using df -h from the guest's OS. Strangely, I am getting Disk Size Full Error. When I checked at the Windows host, the disk space used is shown as 200kb less to reach 350GB. The trash bin, both at the Host and Guest is empty.

Am doing a set of processes that will generate a set of files amounting to 4 GB. Will pre-process the files, keeping less than 200kB of information then delete the original 4GB files. I will keep doing this (generating 4GB, keeping 200kB and deleting the original 4GB) for several days using an automated script.

Tried reading section 5.2 at the Manual but have not got inkling why this happen. Was not successful when searching the archive in this forum too.

I am not sure but my theory to explain this could be as below.
1. Say during the fresh installation of VDI the file size is 10GB, when I generate 4GB of File A, the VDI size now is 14GB.
2. When I delete the 4GB and keep 200kB, the VDI file size remains at 14GB (this is understood as dynamic file allocation does not shrink the VDI).
3. When I generate my second 4GB (File B), rather than using the remaining space of 14GB (allocated during File A), it expands to 14GB + 4GB (File B).

Can you please guide me if the above is the correct operation of dynamically allocated VDI? Does it makes sense from the filesystem point of view or simply a bug?

Feel free to share with me any link if this has been discussed before. It would surely be very useful for me.

Thank you very much for your help.

[mpack]

This is normal behaviour. VirtualBox is not an operating system, it's a hardware simulator, hence in the case of disk simulations it works at a hardware level where files don't exist: only sectors. Disk space becomes used (and the host dynamic VDI grows) when guest software writes to a disk sector that hasn't previously been written to. Once that's done there is no way to "unwrite" a sector using official VBox tools. Hence if you have a guest process that somehow arranged to write to every sector of the virtual disk surface then the dynamic disk would grow to its fully allocated size - 1TB - assuming your host drive has enough space to allow that.

IMHO it is bad practice to create VDIs which are larger than the host drive, if there's a real chance that they'll grow like yours has. Also, you should avoid guest processes which you know will write to the entire (or large chunks of) the virtual disk surface. Examples of this are: secure delete apps, "thorough" disk surface tests, defragmenters can be a problem too.

CloneVDI is a third party tool which can compact your disk again, down to the size needed to contain the actual data - you'll need space on a second drive equal to the compacted size. I'm suggesting this as a way to recover your situation, but in the mid term you should probably stop doing what you were doing to expand the disk.

[mpack]

On the process you describe: if you're truly deleting the data sets as you go then eventually I would expect the guest to start re-using sectors of deleted files, and hence the VDI would reach an equilibrium size far short of 350GB. But, I'm not thoroughly familiar with how Ubuntu manages disk space. However you may want to double check that the files really are being deleted, and not moved to a trash folder, held in a network cache folder etc.

[Saravanan Kandasamy]

Dear mpack,

Yes you are right. I am deleting the huge intermediary files created. My steady state disk consumption at any one time would be way lower than the actual disk size (350GB) though the VDI is configured for 1TB. At 35GB (10%) for the worse case.

The problem I am seeing is, the host is not reusing the emptied sectors but keep on enlarging the disk size. IMHO it would be efficient if the file management re-uses the empty sectors first and only when it's full, it go forward to dynamically increase the VDI size till the maximum size configured.

Do you think this should be addressed by Ubuntu in the way they write the sectors rather than VM VirtualBox?

Note: The VDI is configured to 1TB due to my past experience of getting tired to increase the VDI space size due to legitimate disk full error. It thought I would increase this once and for all and would not see the disk full error unless I am out of physical space. Looks like this is not a clever approach

[mpack]

Do you think this should be addressed by Ubuntu in the way they write the sectors rather than VM VirtualBox?

Yes, that's exactly what I think, though I admit I'm guessing. I don't know much about Ubuntu, but from what you are saying their allocation algorithm may have a preference for previously unused portions of the disk. Perhaps this reduces the potential for fragmentation, or makes file recovery more likely or something like that.

It is definitely not a VirtualBox bug: no VBox bug can control where the guest chooses to write data.

You might be able to address the problem by attaching a 2nd drive with a different filesystem (NTFS say), after compacting your system drive.

[Saravanan Kandasamy]

Dear mpack,

Thanks for your comments.

With your hint on compact, I have read Chapter 8.23 on the manual regarding VBoxManage. It has a command modifyhd that uses the attribute compact. Studied on this further and I bumped to Mark Loiseau's blog whom had provided step by step on how to compact a VDI with Ubuntu as guest.

Refer: http://blog.markloiseau.com/2010/10/how ... -vdi-file/

I followed the steps and the VDI which used to be 350GB now became little lesser than 40GB!

When I thought I can finally celebrate this (despite the process took 18hours to complete as it has to process sector by sector), when I restarted the VM and booted the VDI, it was not able to boot. The screenshot is as attachment.

The booting process was killed, the below is some lines shown on the screen.

Code: Select all

Killed
mount: mounting /dev on /root/dev failed: No such file or directory
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn't have /sbin/init.
No init found. Try passing init=bootarg.

Perhaps I have to remount all those directories manually. (The Blog did not give any hint on this )

Believe it is now a Ubuntu problem rather than VM. How nice it would have been if Ubuntu reuses the empty file sector rather than keep on expanding the VDI on dynamic filesystem.

[mpack]

Sorry, I can't provide support for procedures found in a third party blog.

The "compact" I was thinking of would have used CloneVDI, as I said in my first reply - I also provided a link.