VDI file size keeps growing

[jnb]

My VDI file keeps growing and I don't understand why. I am unsure whether this is a Linux Guest or Windows Host category question. [EDIT: Having typed this question in, I am realising you may say this is a Linux rather than a VirtualBox question, but I've got this far so...]

I am running latest VirtualBox (5.1.18) under Windows 7 (Home Premium) 64-bit as Host. Ubuntu 16.04 (latest patches) 64-bit as Guest. VDI file is using dynamically allocate storage up to 100G.

When I used to run Ubuntu 14.04 all seemed reasonable. Recently I have been finding the VDI image keeps growing in size. It has grown from 7G to 14G. (This may not be much to some of you, but with an 8G memory stick for backup it's a problem for me!) I can boot Ubuntu to Unity desktop and then immediately exit, and the image can grow by 0.5G.

In Ubuntu the output from

df -h

shows:

Code: Select all

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        97G  7.7G   84G   9% /

So that is saying there is an extra 6G beyond Used which has been "lost" compared to Avail.

du -h -x -s /

confirms 7.6G used.

1. What & where is the extra missing storage?
2. Are there known problems here between VirtualBox and Ubuntu 16.04 (14.04 did not seem to have this problem)?
3. I see (e.g. https://superuser.com/questions/529149/ ... -file-size) there are tools like CloneVDI, should I be using that to try to recover space?

[mpack]

It's very simple: if the guest OS writes to a part of the disk that hasn't been written to before, the VDI grows. It's up to you to understand what processes are running in the guest, and why they might be writing to new areas of the disk.

As you say, a drive that grows to 14GB doesn't sound very unusual to me. All my VMs are typically given a 32GB drive, and I regard it as normal for that drive to be half full. Anything very different from half full probably means I chose the wrong size drive.

Possibly it's just a swap area thats being written to: that might happen if you unduly restrict the amount of RAM available to the guest, while not equally restricting the number of programs it has to run.


While CloneVDI could indeed compact the drive, that may be a waste of time unless you can identify the usage patterns that may quickly return it to its previous size.

[jnb]

@mpack Thank you for your reply, as ever!

As I said, while typing in this question I was beginning to realise it looks more like a Guest OS issue than a VirtualBox/host one. It's not that 14G is terrible, it's that all I have to do is fire up, login in & log off to see another 0.5G disappear, lately. There are various posts on the net indicating excessive memory use, swap space size etc. with specifically Ubuntu 16.04 possibly in combination with the desktop, so it's perhaps not a VB-only issue. Which is irritating for me but not for you!

[jnb]

@mpack

Actually I would like to ask you a question about this. I'm interested to understand how it works.

As you presumably know, Linux allocates file storage from a pool of "inodes" (I'm not sure what the deal is for swap space). Given that the VB VDI is "dynamically allocated storage", when you say "if the guest OS writes to a part of the disk that hasn't been written to before, the VDI grows" do we (you) have any idea where the extra space is taken from? Could it allocate an inode pointing to an area of the disk well beyond the current "end of disk", thus using lots of space, or does VB serve the pages out "virtually" so that new allocations will always be located just at the start of the free area?

[mpack]

As you presumably know, Linux allocates file storage from a pool of "inodes"

To be technically accurate, it's the EXTx filesystem which works that way. Linux lets you choose the filesystem, and not all of them use inodes exactly, though they all have some analogue of it.

(I'm not sure what the deal is for swap space).

Nor am I, however I would suspect that it will resemble more a RAM allocator than a system of named file chains.

Given that the VB VDI is "dynamically allocated storage", when you say "if the guest OS writes to a part of the disk that hasn't been written to before, the VDI grows" do we (you) have any idea where the extra space is taken from?

That I can't tell you. It's an implementation detail that could change radically from one version of an EXTx implementation to the next, and I don't really keep up with EXTx variants as it usually makes no difference to anyone beyond the OS implementor. I do have a vague notion (which may be entirely wrong) that EXTx tries to create new files in unused areas of the disk so that they have room to grow without being fragmented: hence you might observe the VDI growing effect if you run an app that tends to create new files instead of reading and/or overwriting existing files.

[arQon]

The most common cause of LARGE discrepancies between "Free" and "Available" is space reserved for the root account, which is usually around 5% but might be something crazy like 20% of the disk space. use "sudo tune2fs -l" to check yours.

On a related note:

You're never going to stop the VDI from growing, but if you want/need to shrink it for backups/whatever, this works nicely:

Code: Select all

~# cat zf
stop rsyslog
service dbus stop

swapoff /swap
dd if=/dev/zero of=/swap bs=1M count=2048
mkswap /swap

if mount -n -o remount,ro /dev/sda1
then
  fsck -f /dev/sda1
  echo zeroing unused space...
  zerofree /dev/sda1
  poweroff
fi

(Change the swapfile handling to match your system, obviously).

Once you have that sorted out, "init 1" will drop the system into single-user mode, and running the script will zero out the empty parts of the VDI and shut down the VM. At that point, "vboxmanage modifyhd --compact <vdi>" will discard all the empty blocks, leaving you with a minimized VDI for backups.

[mpack]

I'm not sure why anyone would want to use that cumbersome (and ineffective in non filesystem areas) compaction procedure when they've already discovered CloneVDI, as the discussion indicates.

[Perryg]

There are actually a lot of reasons why the file size would increase ( none of them due to VirtualBox however ). First swap has nothing to do with your disk space. It is a separate partition ( for now ) And unless you post the results of "fdisk -l" we won't know what it even is. Usually it is 1 1/2 times the amount of RAM that is installed for the OS which is used to move unused running programs to when more memory is needed to run programs/application and should not be considered as an extension to the amount of installed RAM. This can be manually cleared with "sudo swapoff -a && sudo swapon -a" in a terminal or you can simply reboot.

Typical reasons for expanding disk space is updates to programs and or system processes like kernels. These vary from OS to OS as to how they are handled but some OSes play nicer than others in getting rid of older ones when they are updated/upgraded. You can clean these manually with apt-get autoremove followed with apt-get autoclean in an elevated terminal. Another reason I have found is upgrading. If the guest was say 14.04 and you upgrade it to 16.04, all bets are off when it comes to conservation. The installers just don't have the proper knowledge of the entire process to make it effective. Same thing with non-Linux OSes so it is a general issue with upgrading.

Lastly if you do not normally shut the guest down but use saved state the various discardable bits will not be eliminated. This only happens with a system reboot/shutdown.

[arQon]

I'm not sure why anyone would want to use that cumbersome (and ineffective in non filesystem areas) compaction procedure when they've already discovered CloneVDI, as the discussion indicates.

Mostly because I'd never heard of CloneVDI somehow (thanks for it, looks very handy) but also because not everyone is on a Windows host, and cumbersome though it is that procedure works everywhere.

Can you explain the "ineffective in non filesystem areas" part please? Do you mean "if there's an unformatted partition on the VDI"?

[mpack]

but also because not everyone is on a Windows host, and cumbersome though it is that procedure works everywhere.

They don't have to be on a Windows host. CloneVDI works very well under Wine on Linux and OS X hosts.

Can you explain the "ineffective in non filesystem areas" part please? Do you mean "if there's an unformatted partition on the VDI"?

Let's say you shrank the main partition some time after creating the VM (it's a common scenario when someone decides they made the disk too large for comfort). That leaves dirty sectors in unpartitioned areas of the disk. Using dd to zero fill that area will not work, as you can't address that area, but CloneVDI can. There are other examples commercially available.

[arQon]

Wine is often not an option (certainly not for me) but it's good to know regardless.

re dead space, thanks for the example and clarification.