vidma - Virtual Disks Manipulator (tool for resizing VDI)

Discussions related to using VirtualBox on Linux hosts.
przemoc
Posts: 32
Joined: 4. Sep 2008, 16:02
Primary OS: MS Windows other
VBox Version: PUEL
Guest OSses: Linux, Windows, DragonFlyBSD, FreeBSD, NetBSD, OpenBSD
Location: Poland

vidma - Virtual Disks Manipulator (tool for resizing VDI)

Post by przemoc »

vidma is a command-line utility for manipulating virtual disk images.
It can show basic information about the image or resize the virtual disk it contains.

Resizing is done by:
  • creating modified copy of the image (original one is safe as it's not being modified at all) or
  • in-place modification of the image
Usage:
vidma INPUT_FILE                               # show information about the INPUT_FILE image
vidma INPUT_FILE NEW_SIZE_IN_MB                # resize the INPUT_FILE image in-place
vidma INPUT_FILE NEW_SIZE_IN_MB OUTPUT_FILE    # create resized image of INPUT_FILE into OUTPUT_FILE
                                               # (OUTPUT_FILE must be different than INPUT_FILE!)
Resize operation requires your confirmation, so don't panic when you hit Enter without rechecking all the arguments provided to vidma before.

Check also manual page for details on command-line syntax and more.

CAUTION!
Program is in ALPHA stage, therefore may be HARMFUL and UNSAFE!
You have been warned! USE AT YOUR OWN RISK! NO WARRANTY!
To reduce possible damages of in-place operation ALWAYS BACKUP YOUR IMAGE or just do not use in-place operations at all.
To avoid in-place modifications always provide output file in the command line, which must be different than input file.


PLEASE USE ONLY THE LATEST VERSION.

Supported formats
  • VDI - Virtual Disk Image
    Format introduced by VirtualBox and mostly used by VirtualBox. It has a few variants, but only two types, fixed and dynamic, are handled by vidma.
Requirements
  • little-endian machine, e.g. x86, x86-64
  • Windows or POSIX OS, e.g. BSD, Linux, Mac OS X
NEWS
What's new in version 0.0.4 (2012-12-31)
  • Unallocated blocks of zeroes are handled now properly (important for shrinking dynamic images). Bug discovered thanks to Romain Guinot.
  • Non-zero extra blocks are supported now.
  • Upcoming image size change is shown before the resize operation, along with the required free space for the operation and free space that is available on the volume at this moment.
What's new in version 0.0.3b (2011-10-13)
  • Messages printed by vidma have been slightly changed.
  • Simple configure script has been added. You have to run it before invoking make. Building outside of the source directory is now supported.
What's new in version 0.0.3a (2011-02-28)
  • Manual page has been added.
What's new in version 0.0.3 (2011-02-26)
  • Resizing dynamic VDI files is finally supported!
    First expanding will (almost) always move blocks, but next ones will do it only if you cross 255 GB boundary, and again for ~512 GB, etc. In future it will be fixed, i.e. all block moving will be avoided for second and further resizes if block size equals multiple of 1 megabyte (I haven't seen any image "breaking" this rule in the wild, but it's possible).
    Even shrinking is possible, but only if does not involve discarding of allocated blocks. In future it will be also fixed with implementation of block linearization.
What's new in version 0.0.2 (2011-02-22)
  • Many restrictive assumptions about supported VDI files has been removed. Images created in VirtualBox 4.x should work fine now. Still only fixed ones can be resized.
  • Before resizing user gets information about upcoming operation and question whether it should be really performed.
  • Resizing strategy has been changed. Old one was almost always moving blocks, which is simply stupid and inefficient. Now if data offset can be preserved, i.e. there is enough space for block allocation map, block moving will be avoided. If it cannot be avoided, then new data offset will be aligned to megabyte. It nicely prevents from moving blocks in possible future resizes (unless you cross 255 GB boundary).
  • Running in Windows 2000 is possible now.
What's new in version 0.0.1 (2009-10-21)
  • Nothing, it's just the first public version.
License
GNU General Public License v2

Links
git clone [url=git://github.com/przemoc/vidma.git]git://github.com/przemoc/vidma.git[/url]
Repositories: GitHub
Source code: tar.gz archive from master branch
Executables: Windows (32-bit)
Packages: ArchLinux [0.0.3b in AUR], PLD [0.0.4]
Manual page: vidma(1)
Reporting issues: please read Bugs section of README file

Latest release direct links
Source code (tar.gz): 0.0.4 @przemoc.net, 0.0.4 @github.com
Windows 32-bit executable (exe): 0.0.4 @przemoc.net, 0.0.4 @github.com

Walk-throughs
Building vidma straight from git repository and converting 2 GB fixed VDI into 75 GB dynamic VDI (Linux, v0.0.3b+)
Creating 1 GB fixed VDI in VBoxManage and resizing it in vidma (Windows, v0.0.4)

Some details on VDI format
There is so called virtual machine, VM for short. It's the "virtual computer" that you run in VirtualBox (VB). It has many components, just like real computer, some of them are (hard) disks, or virtual (hard) disks if we want to be precise, sometimes called VD or VHD for short. VDI is one of formats of virtual disk images, introduced by VB and mostly used by it. VDI stands for "VirtualBox Disk Image" (or "Virtual Disk Image"). VDI has a few variants like fixed, dynamic and some snapshot related ones (snapshot images are not supported by vidma yet).

VDI consists of 3 areas:
  • preheader + header (start),
  • block allocation map (BAM),
  • allocated blocks (data).
BAM offset and data offset used to be aligned to 512 bytes in the old days, but since VirtualBox v4.0 (released 2010-12-22) they changed it to 4096. It's a good change considering block/cluster sizes of filesystems and sector sizes of drives today. If BAM offset is not 4K-aligned, there is no real gain in fixing it though. And there is definitely no point in decreasing once increased data offset. Even if it's only 1MB, then BAM can map up to (almost) 256K blocks. Take into an account standard block size = 1MB and you get (almost) 256GB. That's why vidma preserves BAM offset and enforces 1 megabyte alignment for data offset, but only if movement of allocated blocks is unavoidable. Forgive my waste.

BAM is populated with 3 types of entries:
  • allocated block (the only one used in fixed VDI),
  • zero-filled (unallocated) block,
  • unallocated block.
Some details on resizing images in vidma
You have to understand that vidma works only at virtual disk level, i.e. it can resize your VDI, but it won't alter its partitions and partition table before or after the change. It's up to you - you can do it using e.g. GParted that is available in the handy Parted Magic bootable live CD.

With vidma you can shrink the VD in-place.

If it's a fixed VDI, there is no restriction upon new size, so be extremely careful. You'll see below warning in such cases. You should perform such operation only when you're sure that it's safe - that usually requires shrinking partitions before shrinking the virtual disk image.
WARNING Shrinking disk in-place means
        IRRETRIEVABLY LOSING DATA KEPT BEYOND NEW SIZE!
If it's a dynamic VDI, then there is one simple restriction: you cannot lose any data - so it's not as flexible as in the case of fixed one, but it's quite safe. But... relying solely on blocks with data (allocated or zeroes) is not enough to be 100% safe, because filesystem format (rarely modifying whole partition nowadays) could simply not modify the last sector of the partition and similarly user's files could not reach that point either yet. In such case you can shrink the image more than you should, making last partition(s) geometry no longer correct, breaking file system size metadata (if stored), and so on... literally asking for lots of trouble. But unless you use such image and make the damage running it under VM, this image is perfectly fine in terms of data stored in it and can be "cured" by enlarging it to the point where last partition ends.

Afterword
I repeat it once again: This tool is not well tested, so always have a backup of image you want to modify in-place. OTOH I hope that VB community will help me here by testing it on wide range of platforms (currently only little-endian machines are supported) and finding bugs.
Last edited by przemoc on 25. Jul 2015, 21:10, edited 23 times in total.
Desktop: NUC11PHKi7C [Core i7-1165G7 @ 2.80GHz, DDR4-3200 32GB RAM, GeForce RTX 2060, M.2 NVMe SSD 2TB] | 32UN880-B
Laptop: ThinkPad T430 (2344-56G) [Core i7 3520M @ 2.9GHz, DDR3-1600 8GB RAM, NVS 5400M, SATA3 SSD 512GB]
Sasquatch
Volunteer
Posts: 17798
Joined: 17. Mar 2008, 13:41
Primary OS: Debian other
VBox Version: PUEL
Guest OSses: Windows XP, Windows 7, Linux
Location: /dev/random

Re: vidma - new utility for resizing fixed-size VDI

Post by Sasquatch »

And what does this actually add compared to the CloneVDI tool from mpack? That one runs on Windows, and when you have Wine, also on Linux.
Read the Forum Posting Guide before opening a topic.
VirtualBox FAQ: Check this before asking questions.
Online User Manual: A must read if you want to know what we're talking about.
Howto: Install Linux Guest Additions
Howto: Use Shared Folders on Linux Guest
See the Tutorials and FAQ section at the top of the Forum for more guides.
Try searching the forums first with Google and add the site filter for this forum.
E.g. install guest additions site:forums.virtualbox.org

Retired from this Forum since OSSO introduction.
przemoc
Posts: 32
Joined: 4. Sep 2008, 16:02
Primary OS: MS Windows other
VBox Version: PUEL
Guest OSses: Linux, Windows, DragonFlyBSD, FreeBSD, NetBSD, OpenBSD
Location: Poland

Re: vidma - new utility for resizing fixed-size VDI

Post by przemoc »

Sasquatch wrote:And what does this actually add compared to the CloneVDI tool from mpack?
It allows you resizing of VDIs, but currently only fixed-size ones. I am not aware of any other tool with such feature. It was the main reason I created this utility.
One important drawback in vidma is lack of UUID generating, so there is no proper cloning. It will be fixed someday.
Sasquatch wrote:That one runs on Windows, and when you have Wine, also on Linux.
My tool runs on:
  • POSIX OSes, it was tested on Linux 32-bit (Slackware 12.1) and 64-bit (Ubuntu 8.10) - there is source code, just download, extract and run make (so there is no need for wine)
  • Windows, it was tested on 32-bit (XP SP2) and 64-bit (XP64 SP2) - there is a native 32-bit binary
Desktop: NUC11PHKi7C [Core i7-1165G7 @ 2.80GHz, DDR4-3200 32GB RAM, GeForce RTX 2060, M.2 NVMe SSD 2TB] | 32UN880-B
Laptop: ThinkPad T430 (2344-56G) [Core i7 3520M @ 2.9GHz, DDR3-1600 8GB RAM, NVS 5400M, SATA3 SSD 512GB]
Sasquatch
Volunteer
Posts: 17798
Joined: 17. Mar 2008, 13:41
Primary OS: Debian other
VBox Version: PUEL
Guest OSses: Windows XP, Windows 7, Linux
Location: /dev/random

Re: vidma - new utility for resizing fixed-size VDI

Post by Sasquatch »

Still, that tool from mpack can resize the VDI, dynamic or fixed. Though after a run, it's likely to be a dynamic if it was fixed.
Read the Forum Posting Guide before opening a topic.
VirtualBox FAQ: Check this before asking questions.
Online User Manual: A must read if you want to know what we're talking about.
Howto: Install Linux Guest Additions
Howto: Use Shared Folders on Linux Guest
See the Tutorials and FAQ section at the top of the Forum for more guides.
Try searching the forums first with Google and add the site filter for this forum.
E.g. install guest additions site:forums.virtualbox.org

Retired from this Forum since OSSO introduction.
przemoc
Posts: 32
Joined: 4. Sep 2008, 16:02
Primary OS: MS Windows other
VBox Version: PUEL
Guest OSses: Linux, Windows, DragonFlyBSD, FreeBSD, NetBSD, OpenBSD
Location: Poland

Re: vidma - new utility for resizing fixed-size VDI

Post by przemoc »

Sasquatch wrote:Still, that tool from mpack can resize the VDI, dynamic or fixed. Though after a run, it's likely to be a dynamic if it was fixed.
No, it can only create new image, which is optionally expanded version of original one. It cannot shrink (at all) and increase VDI size in-place.

EDIT:
Of course one thing must be remembered. This tool is only for manipulating virtual disks, so expanding/shrinking partitions and file systems after/before resize operation is task for the user.
Desktop: NUC11PHKi7C [Core i7-1165G7 @ 2.80GHz, DDR4-3200 32GB RAM, GeForce RTX 2060, M.2 NVMe SSD 2TB] | 32UN880-B
Laptop: ThinkPad T430 (2344-56G) [Core i7 3520M @ 2.9GHz, DDR3-1600 8GB RAM, NVS 5400M, SATA3 SSD 512GB]
Sasquatch
Volunteer
Posts: 17798
Joined: 17. Mar 2008, 13:41
Primary OS: Debian other
VBox Version: PUEL
Guest OSses: Windows XP, Windows 7, Linux
Location: /dev/random

Re: vidma - new utility for resizing fixed-size VDI

Post by Sasquatch »

You really didn't take a good look at it, did you? You can tell it to save the 'new' file with the same name as the original, increase the partition size (shrinking will break the VDI), and some more neat things.
Read the Forum Posting Guide before opening a topic.
VirtualBox FAQ: Check this before asking questions.
Online User Manual: A must read if you want to know what we're talking about.
Howto: Install Linux Guest Additions
Howto: Use Shared Folders on Linux Guest
See the Tutorials and FAQ section at the top of the Forum for more guides.
Try searching the forums first with Google and add the site filter for this forum.
E.g. install guest additions site:forums.virtualbox.org

Retired from this Forum since OSSO introduction.
przemoc
Posts: 32
Joined: 4. Sep 2008, 16:02
Primary OS: MS Windows other
VBox Version: PUEL
Guest OSses: Linux, Windows, DragonFlyBSD, FreeBSD, NetBSD, OpenBSD
Location: Poland

Re: vidma - new utility for resizing fixed-size VDI

Post by przemoc »

Sasquatch wrote:You really didn't take a good look at it, did you? You can tell it to save the 'new' file with the same name as the original, increase the partition size (shrinking will break the VDI), and some more neat things.
Shrinking will not break VDI if you appropriately shink partitions (with file systems on them) before that.
I set destination to the same file as source and it broke my VDI, so are you sure that in-place expanding really works?
I don't say that my program has more features than mpack's one. Vidma just has some features unavailable in CloneVDI. That's all.
Desktop: NUC11PHKi7C [Core i7-1165G7 @ 2.80GHz, DDR4-3200 32GB RAM, GeForce RTX 2060, M.2 NVMe SSD 2TB] | 32UN880-B
Laptop: ThinkPad T430 (2344-56G) [Core i7 3520M @ 2.9GHz, DDR3-1600 8GB RAM, NVS 5400M, SATA3 SSD 512GB]
Sasquatch
Volunteer
Posts: 17798
Joined: 17. Mar 2008, 13:41
Primary OS: Debian other
VBox Version: PUEL
Guest OSses: Windows XP, Windows 7, Linux
Location: /dev/random

Re: vidma - new utility for resizing fixed-size VDI

Post by Sasquatch »

Just read that thread and you know what it does.
Read the Forum Posting Guide before opening a topic.
VirtualBox FAQ: Check this before asking questions.
Online User Manual: A must read if you want to know what we're talking about.
Howto: Install Linux Guest Additions
Howto: Use Shared Folders on Linux Guest
See the Tutorials and FAQ section at the top of the Forum for more guides.
Try searching the forums first with Google and add the site filter for this forum.
E.g. install guest additions site:forums.virtualbox.org

Retired from this Forum since OSSO introduction.
przemoc
Posts: 32
Joined: 4. Sep 2008, 16:02
Primary OS: MS Windows other
VBox Version: PUEL
Guest OSses: Linux, Windows, DragonFlyBSD, FreeBSD, NetBSD, OpenBSD
Location: Poland

Re: vidma - new utility for resizing fixed-size VDI

Post by przemoc »

Ok, my bad. Yes, it converted my fixed-size VDI to dynamic-size behind the scene. But why? I didn't set such an option (because it wasn't there?). And there is no in-place expanding, because old file is renamed to "Original ..." and new one is created.
Desktop: NUC11PHKi7C [Core i7-1165G7 @ 2.80GHz, DDR4-3200 32GB RAM, GeForce RTX 2060, M.2 NVMe SSD 2TB] | 32UN880-B
Laptop: ThinkPad T430 (2344-56G) [Core i7 3520M @ 2.9GHz, DDR3-1600 8GB RAM, NVS 5400M, SATA3 SSD 512GB]
Sasquatch
Volunteer
Posts: 17798
Joined: 17. Mar 2008, 13:41
Primary OS: Debian other
VBox Version: PUEL
Guest OSses: Windows XP, Windows 7, Linux
Location: /dev/random

Re: vidma - new utility for resizing fixed-size VDI

Post by Sasquatch »

Go and ask in that topic. I'm sure mpack will answer your question.
Read the Forum Posting Guide before opening a topic.
VirtualBox FAQ: Check this before asking questions.
Online User Manual: A must read if you want to know what we're talking about.
Howto: Install Linux Guest Additions
Howto: Use Shared Folders on Linux Guest
See the Tutorials and FAQ section at the top of the Forum for more guides.
Try searching the forums first with Google and add the site filter for this forum.
E.g. install guest additions site:forums.virtualbox.org

Retired from this Forum since OSSO introduction.
klaus
Oracle Corporation
Posts: 1110
Joined: 10. May 2007, 14:57

Re: vidma - new utility for resizing fixed-size VDI

Post by klaus »

I like two things about this seemingly redundant project:
  • it is open source
  • it is not windows only
The fact that CloneVDI apparently does not support in-place resizing means it's equivalent to the currently available VBoxManage clonehd functionality, just avoiding to use the VirtualBox storage backends (for copyright reasons I assume). I couldn't find source code. Sure, it has a GUI, but that's something I don't get excited about. It is planned to beef up the virtual media manager in VirtualBox to support all variations of cloning, format conversion, renaming and other misc image manipulation as well. It's just work, which our GUI folks didn't get around yet because other issues were more important.

Since this is open source, it's much more exciting: it's a starting point for integrating in-place resizing into the VirtualBox storage backends. Sure, that's a lot of work, but at least there is something we can work on. First one VDI variant, then all VDI variants, then other formats...

If you think you have a better starting point or if you're willing to contibute in this area, just contact me. Any contribution which brings in-place resizing support to VirtualBox is greatly appreciated. This task can be split in small chunks which should be easy to handle.

The VBoxHDD interface already allows for this feature, it's just the VBoxHDD-Plugin interface which needs to be extended, and the VDCopy code adjusted to handle this case. When that's done we can talk about implementing the stub in the VirtualBox API for this purpose. Don't get me wrong, this is just the final integration step. The new code can be tested much earlier, as there are lots of ways to take shortcuts.
sej7278
Volunteer
Posts: 1003
Joined: 5. Sep 2008, 14:40
Primary OS: Debian other
VBox Version: PUEL
Guest OSses: Solaris, Linux, Windows, OS/2, MacOSX, FreeBSD
Contact:

Re: vidma - new utility for resizing fixed-size VDI

Post by sej7278 »

klaus wrote:I like two things about this seemingly redundant project:
  • it is open source
  • it is not windows only
my thoughts exactly, i'm not in the slightest interested in clonevdi for just those reasons (or the opposite of those reasons, you know!)

also, being commandline i prefer.

i think sasquatch is being a bit harsh.......
Sasquatch
Volunteer
Posts: 17798
Joined: 17. Mar 2008, 13:41
Primary OS: Debian other
VBox Version: PUEL
Guest OSses: Windows XP, Windows 7, Linux
Location: /dev/random

Re: vidma - new utility for resizing fixed-size VDI

Post by Sasquatch »

Harsh might be a big word. I'm only trying to figure out why this would be started if CloneVDI already does the majority of it. I asked mpack in his own topic about the source code, and he said to keep it closed at the moment, but it will be shared some day. If anyone from Sun would contact him, I'm sure he would come up with the source code. It would then be a lot easier to implement it in the VB tree, along with this project for the perfect mix. If his code is anywhere near compatible with the current code.
Read the Forum Posting Guide before opening a topic.
VirtualBox FAQ: Check this before asking questions.
Online User Manual: A must read if you want to know what we're talking about.
Howto: Install Linux Guest Additions
Howto: Use Shared Folders on Linux Guest
See the Tutorials and FAQ section at the top of the Forum for more guides.
Try searching the forums first with Google and add the site filter for this forum.
E.g. install guest additions site:forums.virtualbox.org

Retired from this Forum since OSSO introduction.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: vidma - new utility for resizing fixed-size VDI

Post by mpack »

Wow, you get distracted for five minutes and suddenly everybody is talking about you behind your back! :-)
przemoc wrote:[mpack's CloneVDI tool] can only create new image, which is optionally expanded version of original one. It cannot shrink (at all) and increase VDI size in-place.
As Microsoft likes to say, "that behaviour is by design". IMHO, only a fool makes significant and risky in-place mods without making a backup copy of the VDI first. And if you need to copy the file anyway then what exactly is the advantage of doing anything "in place"?

On the "shrink" feature, this also was not an oversight. My existing interface (and underlying code) is perfectly capable of doing it, were it not for the fact that I put code in there to explicitly prevent it, i.e. to complain if the new size is smaller than the existing disk. I cannot imagine what useful purpose is served by the ability to shrink the virtual size of a disk (as opposed to compacting to reduce the actual size of the image file). IMHO, users often find that they made the disk too small and want to enlarge it. I don't remember reading a single post from a user wanting to reduce the virtual size of the disk. Put such a useless but potentially disasterous feature in the hands of novice users? No thanks!
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: vidma - new utility for resizing fixed-size VDI

Post by mpack »

klaus wrote:The fact that CloneVDI apparently does not support in-place resizing means it's equivalent to the currently available VBoxManage clonehd functionality, just avoiding to use the VirtualBox storage backends (for copyright reasons I assume).
That is pretty much precisely what CloneVDI was intended to be: my image of what VBoxManage clonehd functionality should have been, and with a usable interface. I avoided using VBox code for several reasons: (1) this was a fun project and writing my own code is more fun. (2) reliability (VBox may be present, not present, working, or broken, - CloneVDI will still work. (3) Copyright of course, the only use I made of Sun source code was reading some of it to get the VDI spec straight, as no published spec exists (AFAIK).
klaus wrote:It is planned to beef up the virtual media manager in VirtualBox to support all variations of cloning, format conversion, renaming and other misc image manipulation as well. It's just work, which our GUI folks didn't get around yet because other issues were more important.
Promises, promises. I don't remember a single comment from a Sun participant which indicated that a usable cloning interface was seen as even necessary, never mind a priority. I got fed up waiting.
klaus wrote:it is Windows only
What would be your suggested alternative? In fact when I started my project I had a look at what multiplatform GUIs are in widespread use - for a fun project I was certainly willing to learn another API if that had a more widespread appeal than Win32, and covered all platforms. What did I find? I found that Linux originated GUI efforts are a fragmented mess, but that Wine made Win32 by far the most widely supported multiplatform API in existence.

CloneVDI was designed with a Windows host in mind, that being the only platform that I have long experience with and hence the only platform I can support. Only in that sense is it "Windows only".
Post Reply