VICompactor - Tool to compact VDI files.

This is for discussing general topics about how to use VirtualBox.

VICompactor - Tool to compact VDI files.

Postby Pfeifer » 28. Jun 2014, 02:06

Hello All,

I've just created this tool to better compact Virtualbox VDI files.
The tool works a bit like
Code: Select all   Expand viewCollapse view
VBoxManage modifyhd --compact
. My tool, however, understands the filesystem inside the VDI file, and uses the allocation bitmap to figure out which blocks are really in use, discarding all unused stuff. So it is like the VBoxManage thing, but you don't need to run "
Code: Select all   Expand viewCollapse view
sdelete.exe -z
" or any other zero filling method.
In this firts version the tool only understands NTFS inside the VDI file, but other filesystems are planned (like ext2/3/4, FAT32 ..).

The tool is open source (GPL) and can be found in https://github.com/fmpfeifer/VICompactor

Best regards,

Fabio
Pfeifer
 
Posts: 4
Joined: 26. Jun 2014, 19:42

Re: VICompactor - Tool to compact VDI files.

Postby mpack » 28. Jun 2014, 02:25

My CloneVDI tool already provides all of the features you mention - including your planned ones - and has existed and been supported here since 2009. How does your tool improve on it?
mpack
Site Moderator
 
Posts: 30713
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: VICompactor - Tool to compact VDI files.

Postby socratis » 28. Jun 2014, 09:40

mpack wrote:How does your tool improve on it?

It's open source. Not that I'm the programmer, but I guess there are a lot of people (including you) that could contribute and improve the tool. Perhaps even one day include the CloneVDI and the VICompactor functionality in the VBoxManage toolchain itself.

With the utmost respect for everything that you've done/you're doing,
Socratis.
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
socratis
Site Moderator
 
Posts: 27694
Joined: 22. Oct 2010, 11:03
Location: Greece
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5

Re: VICompactor - Tool to compact VDI files.

Postby mpack » 28. Jun 2014, 12:17

<Topic moved to "Using VirtualBox" since "Third Party Apps" does not allow discussion>

socratis wrote:
mpack wrote:How does your tool improve on it?

It's open source.

CloneVDI is also open source, again since 2009. In fact more so, since CloneVDI is covered by a permissive MIT-like license, not one that has onerous conditions like GPL.
mpack
Site Moderator
 
Posts: 30713
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: VICompactor - Tool to compact VDI files.

Postby noteirak » 28. Jun 2014, 22:37

I personally like the fact the tool is written in Java, it is by nature very portable :) I think it's a good effort to have a library more than anything.
GPLv3 is maybe not the best license tho, especially if the goal is to be included in other software, but that's another discussion.
Hyperbox - Virtual Infrastructure Manager - https://kamax.io/hbox/
Manage your VirtualBox infrastructure the free way!
noteirak
Site Moderator
 
Posts: 5198
Joined: 13. Jan 2012, 11:14
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Debian, Win 2k8, Win 7

Re: VICompactor - Tool to compact VDI files.

Postby Pfeifer » 29. Jun 2014, 02:45

mpack wrote:How does your tool improve on it?


It actually doesn't. I wrote it in my spare time to test my java ntfs parser lib, and announced it here just in case it is useful for someone.
And as I plan to also write an java ext2/3/4 parser lib, I have some extra features planned.
And about the license, I have no problem in changing it.
Pfeifer
 
Posts: 4
Joined: 26. Jun 2014, 19:42

Re: VICompactor - Tool to compact VDI files.

Postby hakito » 1. Jul 2014, 12:57

Is your compactor capable of compacting a diff disk only? CloneVDI can only create a merged version based on a diff disk.
hakito
 
Posts: 6
Joined: 1. Jul 2014, 12:56

Re: VICompactor - Tool to compact VDI files.

Postby Pfeifer » 1. Jul 2014, 13:04

hakito wrote:Is your compactor capable of compacting a diff disk only? CloneVDI can only create a merged version based on a diff disk.


Yes, it is. Provided you inform where to find the parent disks, the compactor can understand the filesystem and figure out which unused blocks are in the diff disc, and discard them. Just to be clear: the compactor must read the diff file and the parent disk (or disks), but will compact only the diff disk. You can repeat this process for each parent if you want.
Pfeifer
 
Posts: 4
Joined: 26. Jun 2014, 19:42

Re: VICompactor - Tool to compact VDI files.

Postby Virtual Boxxy » 15. Mar 2016, 07:08

Has anybody used this, and is it reliable enough when compacting NTFS diff image VDI's (snapshots) ?

I have tried it and it seems to work but I haven't used it for any length of time or verified the results apart from "seems to work". Obviously I'm worried about data integrity. The problems with my vbox file in another thread had nothing to do with VICompactor, i never used it on that machine.

For those who arrive here by google, the only way I was able to run it was by installing Eclipse 4.4, creating a new Java project from the makefile, and then running only the VICompactorFrame.java file from within Eclipse. I have never used Java though so I might simply be stupid. Java 1.8.0_25 in Windows.
Virtual Boxxy
 
Posts: 15
Joined: 13. Mar 2016, 15:05

Re: VICompactor - Tool to compact VDI files.

Postby mpack » 15. Mar 2016, 11:16

"Reliable" and "diff image" are mutually contradictory. Just make sure you make lots of backups and you won't need to worry. (And note that snapshots are not backups in any sense).

The tool described above can't possibly be reliable: as pfeifer makes clear in his last post, it modifies the input file, therefore the result (good or bad) is irreversible - unless you have a backup.

If you resent the disk space used by diff chains then stop using them. Use backups only. Besides which, compacting the last element of a diff chain seems pretty pointless to me. The diff state by definition consists largely of dynamic data, so it'll just quickly grow again.
mpack
Site Moderator
 
Posts: 30713
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: VICompactor - Tool to compact VDI files.

Postby aeichner » 24. Mar 2016, 17:00

Just FYI, VirtualBox already has the necessary infrastructure to support filesystem aware compaction. The problem is that only ext2/3 filesystems are supported because we didn't had the time to add additional filesystems and test this properly because there is no demand from paying customers so far. Thatswhy this option is not exposed through VBoxManage. vbox-img in contrast supports it, look for the --filesystemaware option. If anyone wants to help out with adding new filesystems and testing this will be very appreciated. Below are some pointers to the code the filesystem aware compaction support builds upon along with a short description:

Most of the code for the support is located in our Runtime (IPRT). This is also the place where most of the work is required to add new filesystems.

The first part is the DVM (Disk Volume Management) API which understands MBR, GPT and bsdlabel partition schemes: https://www.virtualbox.org/browser/vbox/trunk/src/VBox/Runtime/common/dvm That one should be pretty complete apart from support for EBR (and maybe LVM later on)

The second part is the code for understanding the filesystems which is located here: https://www.virtualbox.org/browser/vbox/trunk/src/VBox/Runtime/common/filesystem
That one needs the most work, one has to add a new source file for each filesystem and add the filesystem descriptor to https://www.virtualbox.org/browser/vbox/trunk/src/VBox/Runtime/common/filesystem/filesystem.cpp The code for the ext filesystem can serve as a template.

The used structures for DVM and the filesystem code are defined and explained here:
https://www.virtualbox.org/browser/vbox/trunk/src/VBox/Runtime/include/internal/dvm.h
https://www.virtualbox.org/browser/vbox/trunk/src/VBox/Runtime/include/internal/filesystem.h

The code in vbox-img connecting everything is here: https://www.virtualbox.org/browser/vbox/trunk/src/VBox/Storage/testcase/vbox-img.cpp#L1260
aeichner
Oracle Corporation
 
Posts: 157
Joined: 31. Aug 2007, 19:12


Return to Using VirtualBox

Who is online

Users browsing this forum: No registered users and 12 guests