Help: Can't compact vdi

[Soulatial]

When I try to compact my virtual harddisk, I get following error

Code: Select all

Error: Shrink hard disk operation is not implemented!

What's wrong?
I know that this works. I have done this some time before.

[TerryE]

Version 2.1 introduced equal support for VMDKs and VHDs. They haven't got compact working for these so it's temporarily been withdrawn. Yup I know what you are going to say, but it's been said by others. I am sorry to say that you'll need to do a disk to disk copy as I explain in my tutorial All about VDIs.

[Soulatial]

What happens if I use the vboxmanage.exe from an older virtualbox version, will that work or what can happen?

Do you have more Tutorials especially for VHD or VMDK. I'm curious what are the differences / advantages of each.

Last but not least. I'm very interested in writing my own VDI defrag program (in C++), not compacting. Of course your tutorial is good, but a technical document with datatypes (e.g. int or __int64 or float,...) would help me very much. Where can I find a technical doc about the VDI format.
I tried it there http://www.virtualbox.org/wiki/VirtualBox_architecture but didn't find anything.
Am I right when I say (and understood your tutorial correctly) that first you have to compact and then defragment the VDI?

[TerryE]

What happens if I use the vboxmanage.exe from an older virtualbox version, will that work or what can happen?

You would need to install an old version.

Do you have more Tutorials especially for VHD or VMDK. I'm curious what are the differences / advantages of each.

Sorry no.

Last but not least. I'm very interested in writing my own VDI defrag program (in C++), not compacting. Of course your tutorial is good, but a technical document with datatypes (e.g. int or __int64 or float,...) would help me very much. Where can I find a technical doc about the VDI format.
I tried it there http://www.virtualbox.org/wiki/VirtualBox_architecture but didn't find anything.

You have to remember that fragmentation can occur at three levels: on the host FS, within the VDI mapping, and within the guest FS. In my experience (especially on Windows OSs, the VDI fragmentation is the least of these. I've got a defrag written in Perl, which I am going to port to python and publish on the forum.

You need to defrag and zero free-space the guest. before you clone or compact.

[Soulatial]

You would need to install an old version.

I know, but I could do that inside a VM or on another PC and so just gain the vboxmanage.exe and nothing more. So I'd like to know what happens when I have this pure file from the old version and the rest on the system is from the new version without the compacting support.

You have to remember that fragmentation can occur at three levels: on the host FS, within the VDI mapping, and within the guest FS.

Or even at for levels, cause I encrypt them with TrueCrypt. So I have the container file that is also fragmented on my harddisk and can't be defragmented.

Now leave TrueCrypt out. Can you really say that VDI fragmentation is only a minor concern? So suggest you have inside the guest a file system with 4 KB clusters. The VDI file is lying unfragmented on the host FS. Now you have a file with 1 MB in the guest FS, that means 256 clusters. These 256 clusters are spread all over the guest filesystems. That means you have to read 256 blocks in the VDI, that means 256 MB.
Suppose the VDI file is internally unfragmented, now you read 256 MB from the host, you position the drive head once and then you read the next 256 MB, lasting 3-4 seconds on modern harddisks.
Now the VDI file is fragmented, that means you have to perform a lot of seeks on the harddisk. Even the file is unfragmented on the host, its internal fragmentation causes a lot of seeks and so wait time.

Now suppose you have the guest file system defragmented and you want to read a 512 MB file. The VDI file still lies unfragmented on the host FS.
With VDI internally unfragmented you read that data in 6-8 seconds.
But now the VDI is fragmented internally, then you have to seek before each read. In worst case you have to seek 512 times, with 10 ms seek time you have to wait additional 5 seconds.
So I see still a problem in the internal fragmentation.

I've got a defrag written in Perl, which I am going to port to python and publish on the forum.

I know you've written that program, but python is still a interpreted programming language and therefore not very efficient and you need to install a python interpreter, so I prefer C(++).

[TerryE]

Can you really say that VDI fragmentation is only a minor concern? So suggest you have inside the guest a file system with 4 KB clusters. The VDI file is lying unfragmented on the host FS. Now you have a file with 1 MB in the guest FS, that means 256 clusters. These 256 clusters are spread all over the guest filesystems. That means you have to read 256 blocks in the VDI, that means 256 MB. Suppose the VDI file is internally unfragmented, now you read 256 MB from the host, you position the drive head once and then you read the next 256 MB, lasting 3-4 seconds on modern harddisks.

You are correct in saying that the VDI uses 1Mbyte pages, but if you look at the source, if the guest does an N cluster read, then this is mapped directly onto the host file system as an N*<cluster size> read. The only time that this is split into multiple reads is if the read spans a 1Mbyte page boundary.

The only time that full 1Mbyte write occurs is in the case of a dynamic VDI where you are writing to a "virgin page", unless the content of the write is binary zeros in which case it is optimised away. Also reads from non-allocated pages do not create a new page, but instead returns zeros.

Incidentally the new version 2.1 clone algorithm just copies the entire content from the source disk to the destination disk from sector 0 to the last sector. Clearly reads from unallocated pages return zeros and zero writes to unallocated pages are optimised out. Hence doing a clonehd effectiveyl defragments the disk and compacts it .

I've got a defrag written in Perl, which I am going to port to python and publish on the forum.

I know you've written that program, but python is still a interpreted programming language and therefore not very efficient and you need to install a python interpreter, so I prefer C(++).

Sorry but you are wrong here. First Python is compiled into bytecodes which are then executed by the Python RTS and is therefore its runtime comparable in runtime to Java, and hence on a modern PC can run at anything between 1-50M Bytecode instructions per second. The basic loop which copies a 1Mbyte block is about 50 bytecode instructions so even if we take the worst case, this is 1-50 ?S per Mbyte copied. So the Python overhead is truly negligible.

[Soulatial]

Incidentally the new version 2.1 clone algorithm just copies the entire content from the source disk to the destination disk from sector 0 to the last sector. Clearly reads from unallocated pages return zeros and zero writes to unallocated pages are optimised out. Hence doing a clonehd effectively defragments the disk and compacts it .

Thank you. That's a good idea to defrag (when you have the necessary space), but OpenSuse won't boot from the clone. I think its because of the changed UUID.

[soundsliketrouble]

What happens if I use the vboxmanage.exe from an older virtualbox version, will that work or what can happen?

You would need to install an old version.

I don't mind, is that all, would that work ?


And does anyone have an ideea how long till we get the compact function back ?

[TerryE]

Don't know but just run sDelete on your filesystems if NTFS, or use the dd if=/dev/zero of=<mountpoint>/zero bs=4096 then rm <mountpoint>/zero if ext3 etc. then use clonehd and swap disks in the VM. A few extra steps but just as effective. You do need the space for the old + new though.

[soundsliketrouble]

What could I be doing wrong again ?

I tried sDelete with the -c switch. Then cloned, but what I got was the same size. Yet, half of the virtual drive is empty, so this means it didn't work. Then I tried the -z switch, endeding up enlarging the vdi. Went back to try with -c again, cloned, gained 30 MB. (The vd is almost 2GB, yet I only have about 1.2 GB in it...)

[Sasquatch]

A bit of overhead is to be expected. You can always use CloneZilla or similar and copy the data to a new VDI file to try to reduce the size.

[TerryE]

I assume that you are using 2.1 or later otherwise clonehd doesn't compact the disk on copy. If you read my tutorial All about VDIs, you will see that I recommend defrag followed by sDelete. You want to consolidate your free space. It only takes one allocated sector in a 1Mb block to prevent it from being compacted away.

[Soulatial]

I wrote a little Tool to clone the GUID.
So if you want to defragment and compact your VDI in one step, clone the VDI with vboxmanage. Then run my tool with "clonevdiguid.exe orignal.vdi cloned.vdi /del" Then cloned.vdi will get the GUID of orignal.vdi, then original.vdi is deleted and cloned.vdi will be named original.vdi. If you only need to clone the vdi leave away the switch "/del". It does a version check (scans for for hexcode 7F 10 DA BE 01 00 01 00 as described there http://forums.virtualbox.org/viewtopic.php?p=29267) to prevent modfying other files than VDI files.
This program is beta, use it at your own risk. Worked flawlessly for me, however I can't guarantee that it contains no bugs. If you find one, please report.

I hope this program is usefull for someone.

[MasterEvilAce]

Never mind-- failed to read