VICompactor - Tool to compact VDI files.

[Pfeifer]

Hello All,

I've just created this tool to better compact Virtualbox VDI files.
The tool works a bit like

Code: Select all

VBoxManage modifyhd --compact

. My tool, however, understands the filesystem inside the VDI file, and uses the allocation bitmap to figure out which blocks are really in use, discarding all unused stuff. So it is like the VBoxManage thing, but you don't need to run "

Code: Select all

sdelete.exe -z

" or any other zero filling method.
In this firts version the tool only understands NTFS inside the VDI file, but other filesystems are planned (like ext2/3/4, FAT32 ..).

The tool is open source (GPL) and can be found in https://github.com/fmpfeifer/VICompactor

Best regards,

Fabio

[mpack]

My CloneVDI tool already provides all of the features you mention - including your planned ones - and has existed and been supported here since 2009. How does your tool improve on it?

[socratis]

How does your tool improve on it?

It's open source. Not that I'm the programmer, but I guess there are a lot of people (including you) that could contribute and improve the tool. Perhaps even one day include the CloneVDI and the VICompactor functionality in the VBoxManage toolchain itself.

With the utmost respect for everything that you've done/you're doing,
Socratis.

[mpack]

<Topic moved to "Using VirtualBox" since "Third Party Apps" does not allow discussion>

How does your tool improve on it?

It's open source.

CloneVDI is also open source, again since 2009. In fact more so, since CloneVDI is covered by a permissive MIT-like license, not one that has onerous conditions like GPL.

[noteirak]

I personally like the fact the tool is written in Java, it is by nature very portable I think it's a good effort to have a library more than anything.
GPLv3 is maybe not the best license tho, especially if the goal is to be included in other software, but that's another discussion.

[Pfeifer]

How does your tool improve on it?

It actually doesn't. I wrote it in my spare time to test my java ntfs parser lib, and announced it here just in case it is useful for someone.
And as I plan to also write an java ext2/3/4 parser lib, I have some extra features planned.
And about the license, I have no problem in changing it.

[hakito]

Is your compactor capable of compacting a diff disk only? CloneVDI can only create a merged version based on a diff disk.

[Pfeifer]

Is your compactor capable of compacting a diff disk only? CloneVDI can only create a merged version based on a diff disk.

Yes, it is. Provided you inform where to find the parent disks, the compactor can understand the filesystem and figure out which unused blocks are in the diff disc, and discard them. Just to be clear: the compactor must read the diff file and the parent disk (or disks), but will compact only the diff disk. You can repeat this process for each parent if you want.

[Virtual Boxxy]

Has anybody used this, and is it reliable enough when compacting NTFS diff image VDI's (snapshots) ?

I have tried it and it seems to work but I haven't used it for any length of time or verified the results apart from "seems to work". Obviously I'm worried about data integrity. The problems with my vbox file in another thread had nothing to do with VICompactor, i never used it on that machine.

For those who arrive here by google, the only way I was able to run it was by installing Eclipse 4.4, creating a new Java project from the makefile, and then running only the VICompactorFrame.java file from within Eclipse. I have never used Java though so I might simply be stupid. Java 1.8.0_25 in Windows.

[mpack]

"Reliable" and "diff image" are mutually contradictory. Just make sure you make lots of backups and you won't need to worry. (And note that snapshots are not backups in any sense).

The tool described above can't possibly be reliable: as pfeifer makes clear in his last post, it modifies the input file, therefore the result (good or bad) is irreversible - unless you have a backup.

If you resent the disk space used by diff chains then stop using them. Use backups only. Besides which, compacting the last element of a diff chain seems pretty pointless to me. The diff state by definition consists largely of dynamic data, so it'll just quickly grow again.

[aeichner]

Just FYI, VirtualBox already has the necessary infrastructure to support filesystem aware compaction. The problem is that only ext2/3 filesystems are supported because we didn't had the time to add additional filesystems and test this properly because there is no demand from paying customers so far. Thatswhy this option is not exposed through VBoxManage. vbox-img in contrast supports it, look for the --filesystemaware option. If anyone wants to help out with adding new filesystems and testing this will be very appreciated. Below are some pointers to the code the filesystem aware compaction support builds upon along with a short description:

Most of the code for the support is located in our Runtime (IPRT). This is also the place where most of the work is required to add new filesystems.

The first part is the DVM (Disk Volume Management) API which understands MBR, GPT and bsdlabel partition schemes: https://www.virtualbox.org/browser/vbox ... common/dvm That one should be pretty complete apart from support for EBR (and maybe LVM later on)

The second part is the code for understanding the filesystems which is located here: https://www.virtualbox.org/browser/vbox ... filesystem
That one needs the most work, one has to add a new source file for each filesystem and add the filesystem descriptor to https://www.virtualbox.org/browser/vbox ... system.cpp The code for the ext filesystem can serve as a template.

The used structures for DVM and the filesystem code are defined and explained here:
https://www.virtualbox.org/browser/vbox ... rnal/dvm.h
https://www.virtualbox.org/browser/vbox ... lesystem.h

The code in vbox-img connecting everything is here: https://www.virtualbox.org/browser/vbox ... .cpp#L1260