virtualbox.org

Hello,

According to the documentation, Oracle VM VirtualBox can achieve near-native performance of the virtual machine guest system, provided the guest code was written for the same target hardware and computer architecture as the host computer system. https://www.virtualbox.org/wiki/Virtualization

I was wondering about the VirtualBox I/O subsystem performance and created a number of tests shown below. In addition to flushing the buffer cache and setting DIRECT_IO with the dd command, I also used different sizes to test the affect of the kernel buffer cache. The tests are reproducible when performed in the same sequence with little variation (+/- 1 %) on subsequent runs. Running the Host kernel in 32-bit or 64-bit mode did not affect performance.

My conclusion from the tests is that there is a huge performance difference between fixed size and dynamically allocated disks. The tests show a constant difference between 300 - 500 %. Is this normal?

I also wonder why it always takes 3 executions of the same command before the the kernel buffer cache kicks in when the disk is fixed size, but works as expected when dynamically allocated.

The performance of a dynamically allocated disk seems about 2/3 of the performance of the underlying hardware. With a fixed disk however, the performance is 3 times faster.

Perhaps the performance loss of dynamically allocated disks is due to the preallocating (my guess) of disk blocks. Any other explanations for the strange performance findings?

Kind regards!

Test Environment:

Virtual machine:

Oracle Linux 6.3 x86_64, 2048 MB RAM
2 Extra virtual disks connected to SATA controller
host I/O cache and solid-state not enabled.
a) /dev/sdd (fixed size)
b) /dev/sde (dynamically allocated)

Host computer:

Mac OS X Snow Leopard (10.6.8 )
Mac Pro 4.1 (8 Core Nehalem), 24 GB RAM, Apple RAID Card (RAID-5) 6 TB.
VirtualBox 4.2.4



Performance results of /dev/sdd (fixed size)

# sync; echo 3 > /proc/sys/vm/drop_caches

# dd if=/dev/zero of=/mnt/sdd1/testfile bs=4M count=1k
4294967296 bytes (4.3 GB) copied, 56.2638 s, 76.3 MB/s

# dd if=/mnt/sdd1/testfile of=/dev/zero bs=4M count=300 (1st run)
1258291200 bytes (1.3 GB) copied, 15.6361 s, 80.5 MB/s
# dd if=/mnt/sdd1/testfile of=/dev/zero bs=4M count=300 (2nd run)
1258291200 bytes (1.3 GB) copied, 0.392249 s, 3.2 GB/s
# dd if=/mnt/sdd1/testfile of=/dev/zero bs=4M count=300 (3rd run)
1258291200 bytes (1.3 GB) copied, 0.334007 s, 3.8 GB/s
# dd if=/mnt/sdd1/testfile of=/dev/zero bs=4M count=300 iflag=direct
1258291200 bytes (1.3 GB) copied, 8.1513 s, 154 MB/s

# dd if=/mnt/sdd1/testfile of=/dev/zero bs=4M count=1k (1st run)
4294967296 bytes (4.3 GB) copied, 67.3149 s, 63.8 MB/s
# dd if=/mnt/sdd1/testfile of=/dev/zero bs=4M count=1k (2st run)
4294967296 bytes (4.3 GB) copied, 67.5066 s, 63.6 MB/s
# dd if=/mnt/sdd1/testfile of=/dev/zero bs=4M count=1k (3st run)
4294967296 bytes (4.3 GB) copied, 66.8631 s, 64.2 MB/s
# dd if=/mnt/sdd1/testfile of=/dev/zero bs=4M count=1k iflag=direct
4294967296 bytes (4.3 GB) copied, 42.6383 s, 101 MB/s

Performance results of /dev/sde (dynamically allocated)

# sync; echo 3 > /proc/sys/vm/drop_caches

# dd if=/dev/zero of=/mnt/sde1/testfile bs=4M count=1k
4294967296 bytes (4.3 GB) copied, 20.7655 s, 207 MB/s

# dd if=/mnt/sde1/testfile of=/dev/zero bs=4M count=300 (1st run)
1258291200 bytes (1.3 GB) copied, 5.69587 s, 221 MB/s
# dd if=/mnt/sde1/testfile of=/dev/zero bs=4M count=300 (2nd run)
1258291200 bytes (1.3 GB) copied, 1.45271 s, 866 MB/s
# dd if=/mnt/sde1/testfile of=/dev/zero bs=4M count=300 (3rd run)
1258291200 bytes (1.3 GB) copied, 0.352501 s, 3.6 GB/s
# dd if=/mnt/sde1/testfile of=/dev/zero bs=4M count=300 iflag=direct
1258291200 bytes (1.3 GB) copied, 3.68386 s, 342 MB/s

# dd if=/mnt/sde1/testfile of=/dev/zero bs=4M count=1k (1st run)
4294967296 bytes (4.3 GB) copied, 13.6296 s, 315 MB/s
# dd if=/mnt/sde1/testfile of=/dev/zero bs=4M count=1k (2st run)
4294967296 bytes (4.3 GB) copied, 13.8362 s, 310 MB/s
# dd if=/mnt/sde1/testfile of=/dev/zero bs=4M count=1k (3st run)
4294967296 bytes (4.3 GB) copied, 16.1571 s, 266 MB/s
# dd if=/mnt/sde1/testfile of=/dev/zero bs=4M count=1k iflag=direct
4294967296 bytes (4.3 GB) copied, 12.9898 s, 331 MB/s

Other Performance results:

Virtual Machine:

# dd if=/dev/zero of=/dev/zero bs=4M count=1k
4294967296 bytes (4.3 GB) copied, 0.458887 s, 9.4 GB/s

Host System:

saturn:~ dude$ dd if=/dev/zero of=./testfile bs=4096k count=1k
4294967296 bytes transferred in 44.833756 secs (95797624 bytes/sec)

I created several large files to avoid the affect of possible disk fragmentation.

Update: I check again and actually confused the fixed size and dynamcially allocated disk. I corrected the mistake.

Tell me that you didn't do these tests with a simulated timer...

And tell me that you recognize that host space allocation on writes to a dynamic drive is a one time cost and that you factored in the same one-time costs incurred when creating the fixed size drive?

IMHO a little knowledge of the subject beats any amount of blind benchmarking: for example, I know that fixed and dynamic VDIs actually use the same format, the only difference is when the host allocation takes place (i.e. on creation or on demand). Hence there is no fundamental reason why either variant should have better performance. Of course actual performances may vary (in either case) due to fragmentation or congestion of the host filesystem, which is a curable problem.

I used the same timer for all tests. Is there a different time for fixed vs. dynamically allocated disks?

I'm not using those tests to actually proof any kind of "real-time" performance, but to show the difference between fixed size vs. dynamically allocated from within the same virtual machine environment.

As I wrote, I was already assuming that host space allocation might have a negative performance affect. However, I can assure you that all the tests are subsequent tests, meaning the actually space has already been previously allocated. Still the performance difference remains at the very same level.

Fragmentation might be an issue, but all virtual disks are on a separate host partition that is not used for user or system or anything else. So there won't be serious fragmentation.

There may not be a fundamental reason for the performance difference between fixed size and dynamically allocated as you outlined, but then please explain what is wrong with the result. Again, those are subsequent tests that can be reproduced as often as you like.

P.S. "IMHO a little knowledge of the subject beats any amount of blind benchmarking" - That's a cool statement, but does little to make friends.

Sorry about the perceived unfriendliness, but I have a low opinion of benchmarks. IMHO they don't really tell you the important thing, which is **why*** the performance of the candidates differ, and hence why you can expect the result to be representative. If you already know the "why" then you wouldn't need the benchmark. They only serve to confuse those... with no other information to go on.

Since I don't care for benchmarks I will not attempt to explain your results, all I can do is refer you to the basic format facts already mentioned.

I understand. No hard feelings. I agree that system statistics and benchmarks cannot tell the reason for a given performance. I also understand that statistics and benchmarks are not necessarily an indicator whether or not a system is working efficiently as it depends on workload. However, I don't think I'm comparing apples and oranges here.

The performance difference is real and regardless of the actual real data throughput, basic format facts or whatever it may be, one does not need a stop watch to notice the huge difference between using a fixed size and dynamically allocated disk image.

I did not post this message to complain or brag about my performance findings, but to find out if the performance difference is to be expected, whether it was a known problem or if there is any other explanation. I did not expect such a noticeable and reproducible performance difference. Well, what I could still do is to try the same on a different computer system and different virtual machine. I have about 61 different virtual machines installed on my current host.

I'm currently writing an article and wanted to include some facts about VirtualBox vs. host performance. I'm usually not simply rewriting things I read, but rather test and verify everything myself. I can leave out details about disk performance, but I find the current answer unclear and not satisfying.

I checked the same tests again, but this time using a complete different computer (Mac Book 10.8.2) and Oracle Linux 5.7 (x86_64).

What is interesting beside the confirmation of the huge performance drop between fixed size and dynamically allocated disks, is that the issue seems with the fixed size disk. I checked my previous test environment again, and indeed, I confused the fixed and dynamic disks results (I corrected my previous reply).

I also wonder where the data of the dynamic disk is actually being written to.

Please see below protocol.

Pluto:210 OL 5.7 GUI (64) bb$ ls -l *.vdi
-rw------- 1 bb staff 3039907840 Dec 18 19:54 210 system.vdi
-rw------- 1 bb staff 21024768 Dec 18 19:54 dyndisk.vdi
-rw------- 1 bb staff 12884955136 Dec 18 19:54 fixdisk.vdi

# parted -s /dev/sdb mklabel msdos mkpart primary 1 -- -1
# parted -s /dev/sdc mklabel msdos mkpart primary 1 -- -1
# mkfs.ext4 /dev/sdb1
# mkfs.ext4 /dev/sdc1
# mkdir /mnt/sdb1 /mnt/sdc1
# mount /dev/sdb1 /mnt/sdb1
# mount /dev/sdc1 /mnt/sdc1

# dd if=/dev/zero of=/mnt/sdb1/testfile bs=4M count=1k
4294967296 bytes (4.3 GB) copied, 20.8267 seconds, 206 MB/s

# dd if=/dev/zero of=/mnt/sdc1/testfile bs=4M count=1k
4294967296 bytes (4.3 GB) copied, 121.889 seconds, 35.2 MB/s

# df -h | grep mnt
/dev/sdb1 12G 4.2G 7.1G 38% /mnt/sdb1
/dev/sdc1 12G 4.2G 7.1G 38% /mnt/sdc1

Pluto:210 OL 5.7 GUI (64) bb$ ls -l *.vdi
-rw------- 1 bb staff 3040956416 Dec 18 20:02 210 system.vdi
-rw------- 1 bb staff 22073344 Dec 18 20:00 dyndisk.vdi
-rw------- 1 bb staff 12884955136 Dec 18 20:02 fixdisk.vdi

where did it write to?

Let's do this again, overwriting the existing file, and creating 1 additional file

# dd if=/dev/zero of=/mnt/sdb1/testfile bs=4M count=1k
4294967296 bytes (4.3 GB) copied, 23.4653 seconds, 183 MB/s
# dd if=/dev/zero of=/mnt/sdb1/testfile2 bs=4M count=1k
4294967296 bytes (4.3 GB) copied, 20.5057 seconds, 209 MB/s

# dd if=/dev/zero of=/mnt/sdc1/testfile bs=4M count=1k
4294967296 bytes (4.3 GB) copied, 115.656 seconds, 37.1 MB/s
# dd if=/dev/zero of=/mnt/sdc1/testfile2 bs=4M count=1k
4294967296 bytes (4.3 GB) copied, 113.254 seconds, 37.9 MB/s

Pluto:210 OL 5.7 GUI (64) bb$ ls -l *.vdi
-rw------- 1 bb staff 3040956416 Dec 18 20:16 210 system.vdi
-rw------- 1 bb staff 24170496 Dec 18 20:12 dyndisk.vdi
-rw------- 1 bb staff 12884955136 Dec 18 20:16 fixdisk.vdi

Is this crazy? Let's compare the UUID's

Pluto:210 OL 5.7 GUI (64) bb$ vboxmanage showhdinfo dyndisk.vdi
UUID: efa09acf-b7cd-4c94-9334-4c8b1a5677f9
Accessible: yes
Logical size: 12288 MBytes
Current size on disk: 0 MBytes
Type: normal (base)
Storage format: VDI
Format variant: dynamic default
In use by VMs: 210 OL 5.7 GUI (64) (UUID: 907cb289-584d-4d10-920c-64ee60d9c4af)
Location: /Virtual Machines/VirtualBox/210 OL 5.7 GUI (64)/dyndisk.vdi


Pluto:210 OL 5.7 GUI (64) bb$ vboxmanage showhdinfo fixdisk.vdi
UUID: e6b569ea-0f60-4b03-9c50-d8cff684cf5c
Accessible: yes
Logical size: 12288 MBytes
Current size on disk: 12288 MBytes
Type: normal (base)
Storage format: VDI
Format variant: fixed default
In use by VMs: 210 OL 5.7 GUI (64) (UUID: 907cb289-584d-4d10-920c-64ee60d9c4af)
Location: /Virtual Machines/VirtualBox/210 OL 5.7 GUI (64)/fixdisk.vdi

# df -h | grep mnt
/dev/sdb1 12G 8.2G 3.1G 73% /mnt/sdb1
/dev/sdc1 12G 8.2G 3.1G 73% /mnt/sdc1

# ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root 9 Dec 18 18:44 scsi-SATA_VBOX_HARDDISK_VB944b41e2-3c582c5f -> ../../sda
lrwxrwxrwx 1 root root 10 Dec 18 18:44 scsi-SATA_VBOX_HARDDISK_VB944b41e2-3c582c5f-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Dec 18 18:44 scsi-SATA_VBOX_HARDDISK_VB944b41e2-3c582c5f-part2 -> ../../sda2
lrwxrwxrwx 1 root root 9 Dec 18 18:44 scsi-SATA_VBOX_HARDDISK_VBe6b569ea-5ccf84f6 -> ../../sdc
lrwxrwxrwx 1 root root 10 Dec 18 18:51 scsi-SATA_VBOX_HARDDISK_VBe6b569ea-5ccf84f6-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 Dec 18 18:44 scsi-SATA_VBOX_HARDDISK_VBefa09acf-f977561a -> ../../sdb
lrwxrwxrwx 1 root root 10 Dec 18 18:51 scsi-SATA_VBOX_HARDDISK_VBefa09acf-f977561a-part1 -> ../../sdb1

Result:
efa09acf (dynamic) = /dev/sdb1
e6b569ea (fixed) = /dev/sdc1

Questions:

There is a constant reproducible 90 seconds performance drop creating a 4.3 GB file using the disk with "fixed size"!
This can also be confirmed comparing the timestamps of the actual disk image (vdi) file.

For some reason, the disk image of the dynamic disk shows only about 24 MB on the host OS, although it uses 8.2 GB out of 12 GB total size? And even after a shutdown it only shows 24 MB. So where are those 8.2 GB stored?

For some reason, the disk image of the dynamic disk shows only about 24 MB on the host OS, although it uses 8.2 GB out of 12 GB total size? And even after a shutdown it only shows 24 MB. So where are those 8.2 GB stored?

Well you only write zeros to the disk, I would believe this is optimized as only marking the parts that have 0, but not the actual data

This could also explained why the fixed size is actually slower, since it HAS to write the 0 (being random data before).

If you run your test with /dev/random, I am sure your benchmark would tell a whole different story.

That's a very good point! I will try it with /dev/random and post the results.

/dev/random didn't work, but /dev/urandom worked. It was very slow though:

# dd if=/dev/urandom of=/testfile bs=4M count=1k
4294967296 bytes (4.3 GB) copied, 678.919 seconds, 6.3 MB/s

It quite obviously started to use I/O and disk space on the host system. So yes, the reason why the previous test did not show actual disk usage was simply because /dev/zero. Thanks for pointing this out!

Since the random generator could slow things down, I created a testfile on the root volume and used a timed cp to copy the file to the fixed and dynamic allocated disks. For more accurate results I deleted the old disk images and created new ones.

# mount /dev/sdb1 /mnt/dynamic
# mount /dev/sdc1 /mnt/fixed

# sync; echo 3 > /proc/sys/vm/drop_caches
# time ( cp /testfile /mnt/fixed && sync )
real 4m52.914s
user 0m0.077s
sys 0m24.654s

# sync; echo 3 > /proc/sys/vm/drop_caches
# time ( cp /testfile /mnt/dynamic && sync )
real 4m50.726s
user 0m0.143s
sys 0m29.785s

According to the above test, there is indeed no performance difference between fixed and dynamic allocated disks. It was all because of using /dev/zero However, the test does not show if there is any bottleneck. The calculated performance of 14 MB/s as such is not very exiting, however, using the cp command in Terminal on the host system shows only 18 MB/s too. Using the Mac OS X Finder to duplicate e.g. dynamic.vdi shows about 66 MB/s, timed with a stop watch. But that's all a bit like comparing apples and oranges.

I finally realized that I did not install the VirtualBox Guest Additions. However, the test results are identical.

Conclusion:

/dev/zero was the culprit for seeing the huge performance difference between fixed sized and dynamically allocated disk images. I can no longer verify that this is the case using a real file and /dev/urandom. My test did not show the reason for the performance results, but considering the previous reply I have no more reason to doubt that there was a performance difference between fixed and dynamic disk images.

The last tests were done on a MacBook Pro under 10.8.2.

waldorfm wrote:/dev/random didn't work, but /dev/urandom worked. It was very slow though

Now we're talking

waldorfm wrote:So yes, the reason why the previous test did not show actual disk usage was simply because /dev/zero. Thanks for pointing this out!

I have to say that when I saw your 3 GB/s speeds, I just had a hunch... Or just give me some of that raw HDD power!
On a serious note, that kind of speed is typically RAM/Cache speed

waldorfm wrote:/dev/zero was the culprit for seeing the huge performance difference between fixed sized and dynamically allocated disk images. I can no longer verify that this is the case using a real file and /dev/urandom. My test did not show the reason for the performance results

Simply your hardware overall - you are doing a lot of things at the same time, and a lot of writting/reading. Nothing surprising here really, especially on a laptop.

Actually the performance of 3 GB/s for reading data is simply due to the Linux kernel buffer cache, even with the "random" file. Once you exceed RAM, the story is different. See below test of consecutive runs on the Macbook. Memory on the Macbook is 4-5 times slower than on the Mac Pro though.

dd if=/dev/zero of=/dev/zero bs=4M count=1k
4294967296 bytes (4.3 GB) copied, 2.09818 seconds, 2.0 GB/s

# sync; echo 3 > /proc/sys/vm/drop_caches
# dd if=/mnt/fixed/testfile2 of=/dev/zero bs=4M count=300
1258291200 bytes (1.3 GB) copied, 30.3374 seconds, 41.5 MB/s
# dd if=/mnt/fixed/testfile2 of=/dev/zero bs=4M count=300
1258291200 bytes (1.3 GB) copied, 0.873992 seconds, 1.4 GB/s
# dd if=/mnt/fixed/testfile2 of=/dev/zero bs=4M count=1k
4294967296 bytes (4.3 GB) copied, 112.288 seconds, 38.2 MB/s

Interesting, it no longer requires 3 consecutive runs for the buffer cache to kick in on the dynamic volume, which I also tested.

Yes it was the buffer cache, but I was more refering to all your first values - they were just over the top everywhere for the dynamic size.
Which brings my next question : why wouldn't you disable cache for this kind of test?

I'm glad now that the disk performance mysteries have been cleared. Thanks again!

I think it is reasonable to say that the possible disk I/O performance loss inside the virtual machine compared to the host OS is not a serious concern, at least not for simple tests in consideration of the overall workload of the computer.

The kernel does the buffering automatically. I used the iflag=direct with the dd command to disable the buffer cache (DIRECT_IO) and also flushed the buffer cache as shown in my first post. Caching of the disk image by the host in Virtualbox settings is disabled.

noteirak wrote:Yes it was the buffer cache, but I was more refering to all your first values - they were just over the top everywhere for the dynamic size.
Which brings my next question : why wouldn't you disable cache for this kind of test?

I did some tests on the Mac Pro again. This time using a file with random content and including file duplicate tests on the same disks. And using dd with direct I/O avoiding the kernel buffer cache.

dd if=/dev/urandom of=/mnt/sdc1/testfile bs=1024k count=4k
4294967296 bytes (4.3 GB) copied, 667.104 s, 6.4 MB/s

dd if=/mnt/sdc1/testfile of=/mnt/fixed/testfile bs=1024k count=4k iflag=direct oflag=direct
4294967296 bytes (4.3 GB) copied, 50.7529 s, 84.6 MB/s
dd if=/mnt/fixed/testfile of=/mnt/fixed/testfile2 bs=1024k count=4k iflag=direct oflag=direct
4294967296 bytes (4.3 GB) copied, 41.3672 s, 104 MB/s

dd if=/mnt/sdc1/testfile of=/mnt/dynamic/testfile bs=1024k count=4k iflag=direct oflag=direct
4294967296 bytes (4.3 GB) copied, 50.926 s, 84.3 MB/s
dd if=/mnt/dynamic/testfile of=/mnt/dynamic/testfile2 bs=1024k count=4k iflag=direct oflag=direct
4294967296 bytes (4.3 GB) copied, 43.5062 s, 98.7 MB/s

dd if=/mnt/dynamic/testfile of=/dev/zero bs=1024k count=4k iflag=direct
4294967296 bytes (4.3 GB) copied, 15.3192 s, 280 MB/s

dd if=/mnt/fixed/testfile of=/dev/zero bs=1024k count=4k iflag=direct
4294967296 bytes (4.3 GB) copied, 13.7794 s, 312 MB/s


The results are as I would expect now, although showing a little better performance for the fixed image. Perhaps this is due to disk fragmentation. The performance as such is pretty much identical to the results of the same tests performed on the host system.