Using as root server | question towards vdi files additional hdds

[kshee]

Hello,

first I want to thank Oracle and the developers for a great software.
I am using VMs since a long time mostly starting learing to use a desktop linux.
This year I started learing to use Virtualbox for setting up a root server (LEMP) with special features.
Before using Virtualbox I used Docker but I don't want to use docker anymore because I don't have the control over the base Image.
Then I was trying to use VmWare Workstation but it lacks the feature of differential Snapshots. Its not practical to transfer all VM files after a single change to the root server everytime.

So now I am using Virtualbox in this way:
Windows as my operating system for creating the Virtual machines.
Then I shutdown the machine and create a snapshot.
After this I transform the Virtualbox vbox xml file with a xslt transformation to use the new snapshot on a root server. I mainly change the hardware settings of the xml file to get it working on the root server.
After uploading the new vdi snapshot files and the new vbox files to the root server I can run the virtual machine on the server.
So I am able to go back to a snapshot if I have any problems on the root server. Furthermore I can test new Software for the root server offline on my development machine.
Its a great setup I think.

Thanks for reading so far. Now my Questions:

Everytime I copy some files to the VM Machine and creating a snapshot the vdi files get larger and larger even if I delete the files on the VM again.
Thats not practical.
I tried to use the program zerofree and afterwards compact the vdi file. But this is only practical when using snapshots. And it did not work well in my opinion.
How can I get the vdi files to remain small.
Or has anyone another idea for the setup? Perhaps Using a separate hdd for a special partition files?

Thank you.
Best regards.

[mpack]

ts not practical to transfer all VM files after a single change to the root server everytime.

Well, it better become practical or the idea must be scrapped altogether.

There is no snapshot technology which allows you to update one base image and magically all the differential images benefit: what actually happens is one of two things. Either (1) only the base VM is updated, or (2) all of the child VMs are corrupted.

The only technology that allows you update one system and have a bunch of child systems synchronize automatically is using a local update manager, and that doesn't require the use of snapshots in the VM. In fact all that snapshots can do is cause problems since an update manager typically would keep track of which PCs have already received the updates.

Snapshots behave like Windows "restore points". I.e. you can roll back the VM to some earlier state, but you can't push new states from elsewhere.

[scottgus1]

Everytime I copy some files to the VM Machine and creating a snapshot the vdi files get larger and larger even if I delete the files on the VM again.

This is expected behavior. Only compacting the vdi will get it smaller again, and you can only compact while the guest is turned off. It will get bigger again after you compact it.

Compacting a snapshot is really tricky if it can be done at all. And it may only work on the most recent snapshot.

In addition to Mpack's advice that snapshots are like Windows System Restore points and are not designed as transfer mechanisms, they are also not backups. Experienced forum gurus only use snapshots on guests they wouldn't bat an eyelash about if they lost. Snapshots make a guest more delicate and should not be relied on for production machines. Every time you take a new snapshot, the old data in old snapshots is locked and cannot be shrunk (that I know of). Even if you delete a file that was actually stored in an older snapshot, that data will not disappear, even if you format the whole guest disk. The data will still exist and take up space in the old snapshot on the host disk. Each snapshot has the potential to become as large as the original base disk was designed to become if you filled it all up. So you could have terabytes of dead data that you will never be able to reliably get rid of.

How can I get the vdi files to remain small.

Keep the original disk size reasonable, and don't use snapshots on the production machines (the servers you want to be live and usable as much as possible).

Here's a possible suggestion for how to format your server, so you can have that "pop in the new program" action you have been trying to achieve with your snapshots, without the delicateness and bloat of the actual snapshots:

Put two drives in the guest: one for the programs and operating system, and one for all the data the programs work with. On your test PC, use snapshots to your heart's content. When you are ready to assemble an update, transfer a copy of the most recent production guest's data disk as the second disk on your test guest. Take snapshots, run updates, test thoroughly. When you are sure the test guest reflects exactly what you want, use Mpack's CloneVDI (usable on Windows or Linux with Wine) and make a clone of the last snapshot of your test guest's OS drive, keeping the old UUID. Now on the production guest, move the guest's OS disk file to a safe place, then copy in the new OS drive clone from your test guest, using the same file name as the old OS disk file, and start the production guest. Your new updates should be ready to use with your existing data.

Also see Moving a VM and re-interpret it as "Backing Up a VM".

[kshee]

Hello,

Thank you for your posts.

I will make backups.

I am using VMwares Snaphot function since 2000 and I have had problems only once and I think it was a user error.
Is Virtualbox Snapshot feature really so buggy can't believe it?

I have tried clonevdi but it didn't make a big difference regarding the filesize if I use the cloning feature inside virtualbox or this program (using compact option).
Why should I use a third party tool? Anyway thanks for programming and have an alternative...

Best regards.

[scottgus1]

Virtualbox Snapshots aren't buggy. The users are. Snapshot issues reported on the forums quite predominantly resolve to pebkac errors.

re CloneVDI:

Why should I use a third party tool?

Because it works. Well. Very well.

Both systems, Virtualbox and CloneVDI, will achieve the same final size disk files (approximately). The Virtualbox compacting and snapshot-handling systems work on the disk files themselves, and if the host PC has a disk error while the operations are happening (and this does happen), then the disk files are toast, kaput, sold down the river, gone south, cluster-cussed, {insert desired euphemism for "failed and completely unusable"}.

CloneVDI, however, works on a copy of the disk file. The originals are never written to by CloneVDI's normal operations, if at all. Thus host PC errors will still leave you a working guest if the compacting/snapshot-merging fails.

[kshee]

Hello,

thank you for the post.

Is it possible to create a new shapshot run clonevdi afterwards to compact the corresponding snapshot vdi.
Then work with the new snapshot, create a new one later, compact again with clonevdi and so on and so on?

[scottgus1]

Keep in mind that CloneVDI makes a (compacted) copy of your original disk and snapshot chains. (Compacting is not mandatory, there's a checkbox for it.) Your original disk and snapshots still exist, and your guest is still connected to them. The copy can be substituted for the original, or can be put into a new guest. (Note that if a new guest is used, reactivation and licensing may be required in the new guest; and if you use the CloneVDI copy in the old guest you may not be able to go back to the snapshot chain anymore. Be very careful, and take folder-copy backups of the whole guest folder and all files therein.)

You can take a snapshot to be able to make safe changes & roll back if you want. Then once you are sure your changes are good you can either CloneVDI or use the built-in Virtualbox commands to merge that snapshot into the original disk. Note that the correct command to use to merge the snapshot into the previous disk in the chain isn't called Merge. I have not been able to wrap my head around the names of the Virtualbox snapshot commands - they seem not intuitive to me, or I just haven't used them enough. Test first with a dummy guest, and take good notes. Also if you decide to use the built-in commands, take a folder-copy backup first.

The built-in commands leave your guest ready to use immediately, provided a host processing error doesn't hose the guest's disk.

CloneVDI gives you a copy pf the guest disk, leaving your originals intact. You need to manually switch to the clone disk. Test if you can switch back to the final snapshot in the chain - I don't know if you can.

[mpack]

The copy can be substituted for the original, or can be put into a new guest.

I would always put the merged clone into a new guest and deal with any issues that arise, rather than try to put it into a VM that has been using snapshots.