Disk performance issue with shared folders

Discussions related to using VirtualBox on Windows hosts.
NamedNick
Posts: 14
Joined: 10. Apr 2013, 19:06

Disk performance issue with shared folders

Post by NamedNick »

Unfortunately initially I didn't mention that I was working on a shared folders. I never thought of that being such a big bottleneck.

I've to work with a 3rd party PHP framework in my virtual machine and encountered performance issue with the disk storage. I know there are and can be many many factors affecting these things, but please let me explain a few observations I made first.

I did a simple "Hello World" page with this big framework (similar to Symphony, Laravel, etc.) and running it locally on a Linux host it has a deliver time of below 50ms just for the raw HTML. Now, I'm not always connected to that Linux host and multiple developers need their own machines to work on their stuff so our idea to provide them with virtual machines.

The guest: Debian Wheezy 64bit , running Nginx, PHP-FPM, Memcache, Mysql
My host: W7 Pro 64bit, i7-980X @ 3.33GHz, 12GB RAM, Intel X25-M G2 160GB
VirtuaBox 4.2.10

With this setup I measured a time of ~800ms delivering a simple "Hello world" page. My goal is to cut it down to ~80ms. Where did I come up with this number?

The same guest running on a Linux host (on a laptop, a much more less powerful system without SSD) yielded around 80ms. And so did the same guest, exported to OVF and imported in VMWare Workstation 8: it took around 80ms too. To recap, the same virtual machine:
  • on my Windows host, in VirtualBox: 800ms
  • on my Windows host, in VMWare Workstation: 80ms
  • on a Linux mid range laptop, without SSD, inside VirtualBox: 80ms
For another test, on my Windows host running the machine inside VirtualBox, I did the following test: create a tmpfs mount point, moved all the framework/code related files into the tmpf and the request was down to ~80ms. I also re-did all the tests on a different Windows machine with a bit less CPU and RAM (but still SSD) and numbers are the same.

I built the virtual machine with vagrant based on a box I built with veewee and configured it with puppet.

The numbers I'm seeing don't make sense to me. Why would Host=Windows and Guest=Linux have such weird problems? I tried going through the VirtualBox guest machines settings, playing around with CPU count, enable/disable certain performance checkboxes, but I couldn't find anything which would yield me better performance. I'm running against a wall.

Any ideas why I would see such behavior? I understand the help I can get is very limited because of the frameworks blackbox, but maybe my measurements rings a bell with someone would my problem could be.

Thanks
Last edited by NamedNick on 28. Apr 2013, 20:35, edited 1 time in total.
noteirak
Site Moderator
Posts: 5229
Joined: 13. Jan 2012, 11:14
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Debian, Win 2k8, Win 7
Contact:

Re: Disk performance issue with Linux guest

Post by noteirak »

Very first question coming to my mind : did you enable I/o cache in virtualbox for the disks?
Hyperbox - Virtual Infrastructure Manager - https://apps.kamax.lu/hyperbox/
Manage your VirtualBox infrastructure the free way!
NamedNick
Posts: 14
Joined: 10. Apr 2013, 19:06

Re: Disk performance issue with Linux guest

Post by NamedNick »

Yes, it was turned on all the time: imgur com/3Eu XTAp.png
(can't post image because of one day user)
NamedNick
Posts: 14
Joined: 10. Apr 2013, 19:06

Re: Disk performance issue with Linux guest

Post by NamedNick »

Doing some naive performance tests shows this (to spoil it: in favor of VirtualBox).

Command used:

Code: Select all

dd if=/dev/zero of=bigfile bs=1024 count=1024000
Random Linux Host sytem at my hand (just for comparison):
1048576000 bytes (1.0 GB) copied, 2.91425 s, 360 MB/s

The guest system in VMWare:
1048576000 bytes (1.0 GB) copied, 15.3881 s, 68.1 MB/s

The guest system in VirtualBox:
1048576000 bytes (1.0 GB) copied, 4.59993 s, 228 MB/s

I understand linear write doesn't necessarily reflect my real problem (which is actually about reading ...). Any idea what would be a quick nice reproducible test for random read I could run on a few machines for comparison?
noteirak
Site Moderator
Posts: 5229
Joined: 13. Jan 2012, 11:14
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Debian, Win 2k8, Win 7
Contact:

Re: Disk performance issue with Linux guest

Post by noteirak »

There is no particular reason why there would a factor 10 loose in performance from a system to another.
Maybe the other mods would have some ideas of why this behaviour
Hyperbox - Virtual Infrastructure Manager - https://apps.kamax.lu/hyperbox/
Manage your VirtualBox infrastructure the free way!
NamedNick
Posts: 14
Joined: 10. Apr 2013, 19:06

Re: Disk performance issue with Linux guest

Post by NamedNick »

Now, I'm still struggling with this, but the new evidence I collected doesn't suggest it's a CPU issue.

Using xdebug profiling I function too take very long which basically only did computational stuff, handlings arrays, etc. This gave me the hint of a CPU performance issue.

So I tested this (yes, that's ten millions; 7 zeros):

Code: Select all

 time php -r 'for ($i = 0; $i < 10000000; $i++) { sha1("foo"); }'
Guest in VirtualBox:
real 1m0.287s
user 0m1.752s
sys 0m58.536s

Guest in VMWare:
real 0m8.657s
user 0m8.629s
sys 0m0.012s

I then switched to another language, in my case Ruby, and got some interesting numbers:

Code: Select all

time ruby -rdigest/sha1 -e '1.upto(10000000) { Digest::SHA1.hexdigest "foo" }'
Guest in VirtualBox
real 0m17.656s
user 0m17.641s
sys 0m0.008s

Guest in VMWare:
real 0m17.311s
user 0m17.269s
sys 0m0.004s

After some research I found the bottleneck: it's actually xdebug.

But interestingly, it's not because I actively use any aspect of xdebug (debugging, profiling), no: it's simply because the module is loaded in PHP. PHP without xdebug:
real 0m4.328s
user 0m4.304s
sys 0m0.020s

It's still interesting because on VMWare, same setup, PHP with xdebug, there's almost no slowdown.
NamedNick
Posts: 14
Joined: 10. Apr 2013, 19:06

Re: Disk performance issue with Linux guest

Post by NamedNick »

I've found a report with xdebug and commented there too: http://bugs.xdebug.org/view.php?id=765
NamedNick
Posts: 14
Joined: 10. Apr 2013, 19:06

Re: Disk performance issue with Linux guest

Post by NamedNick »

I think I made a terrible beginner error while documenting my problem.

I somehow managed to forgot to mention that all the files I work with, the farmework and everything, is not on a directory within the guest machine but is actually on the host system using the shared folder technique from VB.

Using my previous testing reveals this:

Code: Select all

dd if=/dev/zero of=bigfile bs=1024 count=1024000
On native FS inside guest:
1048576000 bytes (1.0 GB) copied, 3.17891 s, 330 MB/s

In shared host folder inside guest:
1048576000 bytes (1.0 GB) copied, 58.0658 s, 18.1 MB/s

With that information everything is clear now: https://www.google.com/search?hl=en&q=v ... older+slow.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Disk performance issue with shared folders

Post by mpack »

If you have found an answer to your question then IMHO it would be polite to summarise that answer here, rather than giving a link to an offsite web page - which may be broken or changed in future when someone reads this page.
NamedNick
Posts: 14
Joined: 10. Apr 2013, 19:06

Re: Disk performance issue with shared folders

Post by NamedNick »

The answer is that the performance of the shared folders is very poor and thus I wasn't able to share the files my guest webserver needed with the host system.

In the end I put the files for the webserver within the guest off the shared folder and use external software (PhpStorm, WinSCP) to ensure the files within the guest are up2date.

I tried being clever and come up with syncing depending on file change notifications, but changes from the host on the shared folder do no propagate to the guest shared folder and thus further synchronization into the replicated directory within the guest was not possible by this means of automation.
pat1498
Posts: 4
Joined: 28. Jan 2014, 21:23

Re: Disk performance issue with shared folders

Post by pat1498 »

Here is a pretty decent comparison of VM disk performance. mitchellh dot com comparing-filesystem-performance-in-virtual-machines

Sorry, I can't post links yet.... scan someone repost that link appropriately

Clearly... virtual box needs some work. My dev. team stopped using vagrant because of the shared folder performance... maybe we can bring it back with VMWare.. or when the problem in VBox is worked out.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Disk performance issue with shared folders

Post by mpack »

The above discussion involves shared folders, not virtual disks, so someone's naive comparison of virtual disk performance seems even less relevant than usual.
pat1498
Posts: 4
Joined: 28. Jan 2014, 21:23

Re: Disk performance issue with shared folders

Post by pat1498 »

That article is also comparing shared folder performance. Maybe you didn't take the time to read it. VBox shared folder performance is terrible
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Disk performance issue with shared folders

Post by mpack »

No, I didn't bother to read it. IMHO benchmarks are never a substitute for understanding.
pat1498
Posts: 4
Joined: 28. Jan 2014, 21:23

Re: Disk performance issue with shared folders

Post by pat1498 »

Sure... YOU can discard the benchmark but the community can't ignore the fact that there is obviously a performance issue with VBox shared folders. Open source project members and their community should work together to improve projects, instead of playing whatever role it is you are taking on here.

I'd hoped it was some sort of configuration issue... like the I/O caching, but it wasn't. As with NamedNick's setup... disabling XDebug helped, but it doesn't solve the root issue.
Post Reply