Discuss All about VDIs

This is for discussing general topics about how to use VirtualBox.
jerome.hode
Posts: 4
Joined: 29. Jul 2008, 10:33

Post by jerome.hode »

TerryE wrote:Sparse files could be a mixed blessing and curse....
You're surely right.

Anyway, the "sparse experiment" was not meant to provide general-purpose "better" VDI files, but just a space-friendly alternative to fixed ones, in case you need the fixed format for some interoperability (f.ex with imdisk)
TerryE wrote:The sparse attribute is not sticky on copy so cloning (or even copying) a sparse VDI will result in it exploding to full size.
unless you copy it with MakeSparseVDI or MakeSparseFile... :wink:
TerryE wrote:A properly groomed dynamic VDI (e.g. one that has been copied using Gparted, then defragged using the SysInternals Contig tool) will actually run faster than a static VDI. The sparse allocation algorithm is essentially a framgmenting one so that sparse files will be noticeably slower than than static VDIs.
not sure to understand this one.
I would expect a defragged guest volume hosted by a defragged fixed sparse VDI to run rather good.
But I lack experience with both VDI and NTFS sparse files to affirm anything in this domain.
chronoboy wrote:Have you done a benchmark on Sparse files -vs- Dynamic VDI
well, no.
But I would be surprised to see any significant difference in performance between both, when similarly and properly defragmented....

PS: TerryE: thanks for the BBenhancement (I'm new to forum posting)
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

Have look at again my tutorial All about VDIs where I dicuss these fragmentation issues. Basically the blocks in a dynamic VDI are created in chronological order. I have a little Perl utility which does a VDI copy but rearranges them into physical order. Doing a Gparted or Clonezilla copy has the same effect because these only copy non-free space and do so in ascending cluster order. See my discussion about the Sysinternals utilities. Both sdelete and contig are "must have"s for anyone doing their hosting on Windows.

A sorted "contig"ed dynamic VDI runs about 10% faster than a typical static VDI.
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

I've just been answering another post and I realise that there is a subtle difference between the commands
  • VBoxManage discardstate fred
    VBoxManage discardcurrent fred -state
I need to do a few test cases to understand these properly and then I will update the tutorial.
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
Annihilannic
Posts: 5
Joined: 1. Nov 2008, 06:08

Post by Annihilannic »

Kudos for creating this HOWTO TerryE.

I think it would be worthwhile including some info about converting to VDI for example from VMware vmdk's, about which I found a lot of outdated information on referring to qemu and vditool, which appear to be no longer necessary. See this thread.
aquarius
Posts: 69
Joined: 3. Jun 2008, 22:35
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: MS Windows XP

Post by aquarius »

Kudos to the dedication and quality of your work in these forums!
TerryE wrote: ... I realise that there is a subtle difference between the commands
  • VBoxManage discardstate fred
    VBoxManage discardcurrent fred -state
I need to do a few test cases to understand these properly and then I will update the tutorial.
Did you investigate this further? Given the mistakable names for the corresponding gui-functions discussed in this thread I would be really grateful for a little clarification of this chapter with examples including the full comands and relating them to the actions in the gui.

keep up the good work,
aquarius
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

aquarius thanks for the reminder. I've got a couple of other corrections to make as well. I'll try to do this in the next few days.

And thanks for the recognition. Always nice to have :-)
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

Some email comments, thanks to bmwiedemann. I've put them here for audit purposes:
bmwiedemann wrote:Hi Terry,
I read your great tutorial and found some things that might still be improved...
Since the allocation unit on both NTFS and Ext3 file systems is 4096 bytes, you will therefore get slightly better performance (typically a few %) if you make your VDIs (1024N – 128) MByte long.
You need to remember that partitions often do not start on 4k boundaries - on both host and guest partition tables. e.g. hda1 often starts at sector 63. So as often, YMMV.

Code: Select all

vm4:~# fdisk -lu

Disk /dev/hda: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders, total 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1   *          63     3887729     1943833+  83  Linux
Remember that this dead space will only take up 4 bytes per Gbyte.
should be 4 bytes per Mbyte?
or 4 Kbytes per Gbyte?
This means that you can not only share mutable drives across VMs[...]
should be immutable drives?

and another comment on performance:
I like adding the "noatime" mount option to both guest partitions and the host's partition that holds my VDI files. Otherwise every read operation in the VM will cause a write (or even several writes in case of guest inode entries being updated) on the physical disk which is especially bad for the lifetime of flash media, but also hurts performance.

Ciao
Points 2 and 3 agreed.
Point 4. Hummn, good point. Linux write performance can be quite slow. It may be worth doing some timing tests with and without noatime to see if this has a material performance hit.
Point 1. You are correct in what you say but I don't think that this is material. Since most HDDs are LBA'ed, it is isn't clear what the true track alignment is on modern disks. The 39 sectors is just a historical convention. My real concern here is that the guest OS will typically flush whole clusters in its virtual file system to its virtual disk. You want this guest clusters to be cluster-aligned to host file system otherwise you will fragment these writes on the host OS. IIRC, this does make a performance improvement but its typically less than 10%. Your noatime suggestion might have a more significant impact.

I have a bunch of changes that I want to make to this tutorial. Some are just small errata, but one thing that does crop up quite often is that many users are put off my the geeky intro. Maybe I should move the heavy stuff to the back. It might also be worth splitting this into separate tutorials covering VDI internals, duplication and backup, etc.
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
bmwiedemann
Posts: 32
Joined: 23. Mar 2008, 06:41
Location: $HOME
Contact:

Post by bmwiedemann »

TerryE wrote:Point 1. You are correct in what you say but I don't think that this is material. Since most HDDs are LBA'ed, it is isn't clear what the true track alignment is on modern disks. The 39 sectors is just a historical convention. My real concern here is that the guest OS will typically flush whole clusters in its virtual file system to its virtual disk. You want this guest clusters to be cluster-aligned to host file system otherwise you will fragment these writes on the host OS. IIRC, this does make a performance improvement but its typically less than 10%. Your noatime suggestion might have a more significant impact.
I did not refer the real physical layout of the disk but was rather elaborating on the point you make about mis-aligned 4KB chunks.

Let us first state the problem:
What happens in the ideal/aligned case is: guest writes 4KB to its virtual disk which happens to become an aligned 4KB write in the VDI file and thus a single 4KB write on the host-OS's filesystem which means no IO-overhead.
The un-aligned case happens 7 out of 8 times when choosing alignment at random (because 4KB = 8 sectors of 512 bytes).
In that case, when the guest writes 4KB, the host might need to read 2x4KB, update the middle part and write back 2x4KB to disk which (depending on read caching and write buffering) might give some performance penalty.

When the guest OS writes a 4KB block to its (virtual)disk, with the partition hda1 often starting at sector 63 (32256 bytes offset) which is not a multiple of 4KB because of the guest's partitioning, and you have the VDI header size that is a multiple of 4KB... you still get the performance penalty. The overall alignment is what matters.
TerryE wrote:I have a bunch of changes that I want to make to this tutorial. Some are just small errata, but one thing that does crop up quite often is that many users are put off my the geeky intro. Maybe I should move the heavy stuff to the back. It might also be worth splitting this into separate tutorials covering VDI internals, duplication and backup, etc.
I liked having such a place where I can read those details without sifting through the source-code. You could change the order, though and put the "how to" questions first (about compaction,performance,resize,backup)

I have another comment on in-place size reduction of dynamic VDIs.
5.
You then need to use a utility to zero the free space within the file system (remember that in general free space will not be zero, because it will contain data blocks that have been used and recycled to the file system). This utility must understand the structure of the file system and in particular its free-space maps
I usually use (on both Linux and cygwin) dd if=/dev/zero of=/zero bs=64k ; rm /zero (and possibly similar to /var or /usr or whatever separate filesystems you have) to reach the same effect without going through the hassle of remounting and using special tools. The trick is that the OS's filesystem driver understands perfectly the structure of the file system and in particular its free-space maps.

I now did actual benchmarking to compare atime/noatime performance, but found no significant improvement. 2-4 seconds difference for 17 minutes of compiling a 2.6.27 kernel might well be caused by random noise. Of course, there are more IO-intensive tasks...

Ciao
Bernhard M.
I am who I am
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

Bernard, I was going to disagree on your alignment analysis, but I've twigged on the point you are making. Even though both NTFS and Ext3 default to a 4K cluster size, that is the disk is divided into 4K blocks and large files are allocated an integral number of clusters, the point that you make is that for random historic reasons fdisk etc by default aligns individual partitions on a "track" boundary so it adopts multiples of 63 sectors by default which is coprime to 4Kb. There is not logic to this since most disks now use LBA rather than CHS addressing and disk do not in general have a physical track size of 63 sectors. There is nothing to stop you making your partitions 4K aligned and then everything will line up.

A small problem with the dd if=/dev/zero approach is the root reserve means that you need to do this from root. I just prefer the zerofree approach. :-)
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
davelbarton
Posts: 17
Joined: 14. Nov 2008, 01:16
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Windows XP

Troubleshooting guide?

Post by davelbarton »

First of all, kudos for a lot of great work. However, if you don't mind a newbie suggestion, a "Troubleshooting" section would be a great help to newbies such as myself. I read through the entire thing and did not get a clue about how to fix my problem (posted elsewhere; this is not a problem forum).

But, with or without, thanks so much for writing this! I know how much effort goes into it.
baf
Volunteer
Posts: 829
Joined: 27. Sep 2008, 06:18
Primary OS: Mac OS X Leopard
VBox Version: PUEL
Guest OSses: linux,xp,win7
Location: Luleå or Skellefteå, Sweden

Post by baf »

TerryE wrote: A small problem with the dd if=/dev/zero approach is the root reserve means that you need to do this from root. I just prefer the zerofree approach. :-)
Also a problem with this is that the allocation tables won't be zeroed but I think that zerofree does that too.
bmwiedemann
Posts: 32
Joined: 23. Mar 2008, 06:41
Location: $HOME
Contact:

Post by bmwiedemann »

of course the rm /zero should do the zeroring...
IMHO, if allocation tables would not be zeroed, the disk space would not count as free... it might be different for some file-systems, but at least the ext2/3 block-free bitmap and the FAT cluster allocation table should work this way.
I am who I am
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

Bernard, I don't disagree with what you say, but I just tried doing a zero as you suggest on a partition that was formatted with a mkfs.ext3 using default settings. The sudo dd errored with a No space left on device and a df showed that it was 100% allocated. rm it and the allocation went down to 3%. Great.

However, when I looked at the size of the dynamic VDI, it was now 40% the size of the capacity. Why 40% not 5% or 100%? The reason for not 100% is because the VDI driver has a little optimisation whereby if you write a zero block to a unallocated 1Mb tile in the VDI, the write is optimised away. However all the inodes are touched during the write with non zero content, so any 1Mb tiles containing inode blocks are allocated. OK doing a modifyvdi compact then recovered half of this.

However, since zerofree directly writes to any unallocated clusters which are non-zero you side-step this growth problem entirely.
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
TerryE
Volunteer
Posts: 3572
Joined: 28. May 2008, 08:40
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu 10.04 & 11.10, both Svr&Wstn, Debian, CentOS
Contact:

Post by TerryE »

One of the forum members PMed me with a Q on this topic, it is better to as questions here so that others can read the dialogue :-)
champagj wrote:Your tutorial has been of great help. Specifically in trying to resize vdi's. I qualify as a "non-expert" and followed your recommendations to no avail.

I'm running VB 2.1.0 on a Linux Host. I have a couple dynamic vdi's that I am trying to expand. vdi are stored on a ext2 partition and my VM's partitions are FAT32 (I know, I should go to NTFS but not enough space to run convert...)

The copy process using gparted Live, Disk Copy or simple dd command all seems to copy the partition properly. However I have not try to increase the size of the new partition since I cannot make the new partition to boot.

I got everything from Boot error, hanging screen and NDLR missing errors. In recovery mode I have tried various combinations of fixboot and fixmbr.
TerryE wrote:The tedious bit is then getting the new partition to boot properly ... Once you’ve cracked this for your OS, you can quickly get this down to a routine process, but there are enough complications here to merit a separate topic in its own right."
Questions:
  • Do I make the new partition active before or after the copy?
  • Should I format the new partition before the copy?
  • If I reboot after the copy process, the new partition gets assign the "d" drive.
  • Unclear about the fixboot fixmbr. You say in the tutorial to just run fixboot. Does the order matters?
Any help would be greatly appreciated.
OK. Let's pick this apart. (I am assuming that you are running XP as your guest). First a couple of points (i) you should really do a tunefs -j to convert your ext2 partition to ext3. There are tiny performance advantages to ext2 at a loss of a risk of file system integrity. (ii) I am not sure why you can't run convert32. If necessary make space by moving data onto another (temporary) VDI which you've formatted using the XP disk management function.

With XP I strongly recommend using one partition to store your OS, core Program Files and your Documents and Settings for the Administrator account only. Move all other Documents and Settings including All Users onto a second D partition (this involves some registry tweaks in the case of XP Home, or an easier approach is to use one of the SysInternals tools). All of the partitions apart from the system C partition can be simply copied using file manager simply by attaching the new disk to the old system, logging on in Safe mode and using xcopy to copy the entire contents of the partitions one-by-one from the old disk to the new. This way the new partition can be NTFS and the old FAT32. No problem. In fact this approach also works if you can boot from a BartPE ISO and various other permutations like temporarily cloning the old VDI and copying from the clone to the new ...

Just remember to clean up your C drive before copying it if you are using a clone method rather that a filesystem to filesystem method.

But to your specific Qs:

(i) Read the wiki article on MBR. Making a partition active just sets a bit in the partitions entry in the partition record in the MBR. You can do this at any time. However you should have one and only one active partition on a HDD volume.

(ii) If you are copying a file system using GParted or the like then you don't need to format the destination FS. If you are using XCOPY within XP then you do.

(iii) You have two VDIs attached during the copy. You need to remember to release them from the VM and attach the new VDI in the place of the old one so that when you boot it assume the same drive letters. Also remember that there is nothing stopping you splitting your system over multiple VDIs: one for the C partition, one for the D etc. and in many ways this makes managing your VM a lot easier.

(iv)boot. See as follows.

You will be booting your XP system using a standard MS boot process. Here the MS MBR bootstrap scans the MBR looking for the active partition. For XP this is a fixed bit of code which is place onto the MBR by fixmbr (or infact with most Linux distros you can apt-get it and put it onto a floppy image to load from your liveCD, and some liveCDs have it preconfigured).

The MBR bootstrap then jumps into the bootstrap stored in Sector 1 of the active partition. This is the start of the NTLDR. The NTLDR understands NTFS etc. It processes the boot.ini menu, loads any SCSI drivers if needed, the NT Kernel and HAL. It then reads enough of the registry to select the appropriate H/W profile and loads any device drivers. From there the booted NT kernel takes over and you are away.

Fixboot repairs the integrity of the bootstrap within the boot partition, that is the NTLDR etc because this has some hard-wired offsets in it. This is why you need to specify a drive letter for this. Normally if you are doing a clone of a partition then you won't need to do a fixboot.
Read the Forum Posting Guide
Google your Q site:VirtualBox.org or search for the answer before posting.
champagj
Posts: 2
Joined: 10. Jan 2009, 03:00

Resizing vdi

Post by champagj »

Terry,

Thank you for your prompt eply.

I'll start by saying that I was finally successful (after 3 days..) in resizing the vdi's.

As you indicated in your tutorial, once you got it down to a process it is easy to do. Looks like what "made it happen" was to keep the new vdi as "primary slave" after disconnecting the old vdi. I definitely had to boot with the XP cd to do a fixboot (as you indicated).

The reasons behind my choice of files systems:

I'm running VB on an Aspire One laptop running Linpus Linux. The actual vdi is stored on removeable media (16G SD card).

By choosing ext2, i reduce the # of read/write which are reducing the life of flash memory.

Also, it looks to me that the XP guest running on FAT is faster than NTSF and I suspect it is because of the slow writing speed on flash.

The end result is a decent Linux/XP machine at 2lbs with 4hours battery time with the ease of pulling the card out and "move" the guest to a different host...


On a more philosophical note, I noticed your promotion of multiple partitions versus massive ones. I briefly looked into this option through the XP Pro user account setup but could not find a way to point the "Doc & Sett" directory to a different partition. Am I on the right track? Can you point me to posts that would go over that process?


Once again Terry a big thank you for such dedication to the community. I'd love to contribute answers but it looks like for the next 10 years I will be contributing questions only!

I'll offer to be your French-English translator if you need!

JF
Post Reply