Drive size

[drewex]

Hello all,
About a month ago I did a compacting on my VM and shrunk down a lot. Now when I check my drive 49.1gb full but the file is now 69gb. I will do the compact again but not sure whats causing the regular file size increase?

[socratis]

A VDI is a collection of sectors that have been used at some point or another in your guest. A VDI does not know if a sector is still being used or not. The only way that it knows, is if the sector contains zeroed data.

An OS allocates sectors as it writes files. Be it temporary data files, cache files and so on from the guest, sectors are allocated all the time.It does not zero-out the sectors when a file is deleted, a note is simply made that this sector is not used. That note is done internally, in the OS, not externally in the medium. So, VirtualBox cannot possibly know if a sector is used or not, therefore it remains allocated. The only way to clean the mess is to tell the guest OS to specifically go out and zero-out the freed sectors. That will allow the VDI to clearly know which sectors to discard.

[drewex]

that make sense, thx for the info, i am assuming there is no way to automate this right, currently i got few steps i take which seems a lot of work but might be the only way. Backup (not mandatory but i do it), defrag (probably necessary to get most of the space, right), zero empty space, shut down guest os, then compact

[BillG]

And it is not really a VirtualBox problem. The same thing will happen on a physical disk (which is not surprising, because the OS does not know whether the "disk" is physical or virtual). It will also vary form one OS and another, depending on how aggressively they scavenge space.

I find the best way to compact a .vdi is to clone it using clonevdi setting the compact while copying option, then rename the .vdi files so that you are using the compacted one. You can then use the "old" file as your backup.

[drewex]

interesting so you are saying, the compact tool is better then the built in one? this was what i do to compact. VBoxManage.exe modifyhd "C:\file.vdi" --compact

I am using a backup software that does differential backups so I don't waste space. Also I got an initial backup of the vdi.

[socratis]

I am using a backup software that does differential backups so I don't waste space

I'm not sure if this incremental backup is going to work (as you imagine it would) with a VDI. One small change in the VDI, even an infinitesimal one, will modify the VDI as a file. So your incremental backup is going to be backing up the whole VDI over and over again. That's not a bad thing, it's simply not a space saving operation. On the contrary...

[scottgus1]

Re backups: I run our office domain controller/file server on a Virtualbox guest, about 280GB in size. Once a week I do a full guest shutdown and file copy of all guest files on the host drive a la reinterpreted Moving a VM, to two other strong workstations in the office, and over a couple days, rsynced offsite to my boss's and my houses through a peer-to-peer VPN.

During the week I run in-the-guest backup software to full-backup the email server files once a day and the project work files incrementally once a day and differentially every half hour. The in-the-guest backup software copies the full, incremental and differential backups out of the guest via a network shared folder onto the host, then launches a script on the host to copy the backups to the two alternate-host workstations and offsite to my and my boss's houses.

At any point the office would suffer a loss of only a half hour's computer work, even if the whole building was destroyed. (I admit I am not covered against nuclear attack or major natural disaster, because my and my boss's houses are all in the same town, but I think I would have bigger thoughts than the workplace in such situations. And if my boss wants to set up another offsite backup at a relative's house cross-country, I could do that too...)

[drewex]

I am using a backup software that does differential backups so I don't waste space

I'm not sure if this incremental backup is going to work (as you imagine it would) with a VDI. One small change in the VDI, even an infinitesimal one, will modify the VDI as a file. So your incremental backup is going to be backing up the whole VDI over and over again. That's not a bad thing, it's simply not a space saving operation. On the contrary...

The backup software is inside the vm os and also on the host os but with ignoring snapshots and image files

[drewex]

related question for defragging the guest os drive, If i cannot move the some of the files to the beginning of the drive would compacting move them if i do the sdelete on the empty space. i bet it doesnt.

the volume information is dead center on the vm drive and cannot be moved with any defragmantion software i found, maybe there is a way i just dont know it.

de.png (26.28 KiB) Viewed 4414 times

I could just delete the restore points since i am getting backup done, but seems another layer of protection for me.

[scottgus1]

If i cannot move the some of the files to the beginning of the drive would compacting move them

I don't think so. Compacting, if I remember right, just deletes empty Virtualbox blocks from the drive file. It doesn't change the addresses where the guest OS looks for the data within the drive.

I'm not sure, but I think Sdeleting is old school now: it's not necessary anymore, especially if you use Mpack's CloneVDI.

ignoring snapshots

If your host backup is ignoring snapshots then you're not getting all the data your guests contain, and you may not fully understand what a Virtualbox snapshot is. VB Snapshots are similar to Windows' System Restore points: they are point-in-past-time markers, not extractable, useless without the base system in place, but easier to corrupt because the files are accessible on the host drive. They do not store "files", like a backup folder would. Virtualbox snapshots store changed disk sectors, which may or may not contain the entire file. If you only back up the base vdi, then when you restore that guest, if it starts at all, you will only have the state of the machine before you made the snapshots.

Snapshots make a guest more delicate and do not work as backups. They should only be used on guests you're experimenting with and with data you wouldn't mind losing. (The forums are replete with users destroying their important data because they did something wrong with a snapshot.)

[socratis]

Defraging is one thing, because you're going to consolidate files that are spread across different sections of the HD, although I don't think that the overall number of occupied sectors is going to be different. But definitely the location of the sectors in the HD is not an issue.

[drewex]

If i cannot move the some of the files to the beginning of the drive would compacting move them

I'm not sure, but I think Sdeleting is old school now: it's not necessary anymore, especially if you use Mpack's CloneVDI.

Man I guess I was using an old tutorial. I was wasting time with sdelete. Ok Last question on this then which i will research, can clonevdi do compact without copying the file again.

I understand the snap shots. only use it for one VM that doesnt really require backups, but .SAV is created whenever I save state the VM which I do often, and the files do not require to backed up on the host.

I do not backup VM files directly on the host. I created a backup of them initially when I first setup my VM up. Then I use macrium reflect to my normal backups with full image/ incremental/ differential backups.
This gives me a file based/image base backup of the VM data. In any case the VM dies (even if the files dies) I can recover 2 methods, initial try would be boot up reflect recovery (in the VM), if any case that fails, recover the VDI from original backup then restore VM backup from reflect.

And the host by itself has its own backups image/inc/def but excludes image since they are already backing up.

backing up the VDI would be pointless regularly that is just a huge file.
my current backup scheme besides the initial full image backup / once a month full backup/ weekly differential/ daily incremental. Giving me a restore point to any time in the past month. and since they are file based its a lot smaller just to grab the changes.

Defraging is one thing, because you're going to consolidate files that are spread across different sections of the HD, although I don't think that the overall number of occupied sectors is going to be different. But definitely the location of the sectors in the HD is not an issue.

I did not know this. Thanks. I will stop using the old method and switch to cloneVDI.


Ps: you guys are awesome with all the responses. I learned a few new things that will be helpful.

[scottgus1]

If compacting in place fails during the process, then you have a possibly dead drive file. Compacting to a copy is safer. CloneVDI only compacts to a copy.

Not sure I fully understand your backup routine, but whatever works. Just a bit of advice I picked up once: A backup you can't restore isn't a backup, it was just exercising the computer . Be sure your setup works during restore.

[drewex]

Not sure I fully understand your backup routine, but whatever works. Just a bit of advice I picked up once: A backup you can't restore isn't a backup, it was just exercising the computer . Be sure your setup works during restore.

Its worth while doing a restore and using it for sure. To be honest havent done it in a while. And good point the failed compacting could be huge waste of time if its fails doing the recovery process would suck.

[drewex]

jusst did the compactvdi 76 to 52gb NICE!!! ok and it was pretty cool to see the progress over a command prompt window.