Really miserable i/o and network performance

[Lapsio]

I'm trying to host virtualized storage server supporting 12 TB encrypted software 8 bay RAID6 and performance is awful. Guest Linux can't even keep up with raw 1.3 GB/s disks sequential read achievable on bare metal (CPU bottlenecks on around 750 - 950 MB/s depending on how many concurring VMs are running at the moment, whereas on bare metal stays below 10%), And that's without encryption of course (I had to provide already decrypted drives from host device mapper to VM).

Actual array (with fs) has only around 250 MB/s sequential read (over 500 MB/s on bare metal). After exporting over NFS to other VMs seq read drops to 120 MB/s and random i/o doesn't exist.

I tried various settings: changing CPU core count - anything above 1 core results in OHMYGAWD level pings from guest to host like above 1ms. Considering there's 0.2 ms between physical servers separated by 10G router with firewall, it's pretty terrible result for loopback traffic I'd expect something like 0.01 ms. I tried toggling Nested paging, PAE/NX, VT-x in all combinations, adding more RAM, using jumbo frames, adjusting NFS chunk size, paravirtualized NIC, SCSI/SATA/SAS controllers - nothing helps. Performance drop reaches around 70-80% in seq and A LOT more in random i/o.

[socratis]

Anything reproducible for us to try out? A VBox.log.ZIP for the VM would also help a lot...

[Lapsio]

welll I think it'd be hard to reproduce storage issue without hardware. Maybe it could be emulated by having multiple SSDs and create eg 4 drives on each SSD. If VBox bottlenecks on ~750-950 MB/s two 550 MB/s ssds should be capable of emulating that 1.3 GB/s array more or less. VBox logs, sure logs and .vbox file in attachment.

It contains logs from 2 workstation one with OpenSUSE 13.2 and one with OpenSUSE Leap 42.2.

However networking latency should be easy to reproduce as I actually experience it on all VMs. I posted also my network config in that archive. latency is quite high (~0.3 ms) which is really high for NFS standards. When VM has more than 1 vcore it jumps up to 8ms!!!. As you can tell from network config I also installed VMWare Player for test and it doesn't suffer from i/o issues. Actually CPU load under VMWare is kind of similar to bare metal (~10% cpu usage with full 1.3 GB/s throughput). Initially i wanted to go for QEMU+KVM but VirtualBox is incompatible with KVM so... well.... it's really unfortunate.

EDIT: attachment

[socratis]

Please attach that file locally. See the "Upload attachment" at the bottom of the form.

[michaln]

Two general notes. One, the usage scenario described here is so far from typical VirtualBox use that it might as well exist in a different universe. That makes it uninteresting in practical terms. Two, when lowest latency and highest bandwidth is required, virtualization is a problem, not a solution.

[Lapsio]

As I don't buy "hurr durr not typical use case" I decided to make more benchmarks and comparisons and maybe start some small discussion about why I think this issue points out few serious flaws in VBox. I started from installing full VMWare Workstation 12.5 for comparison and tested more combinations and network configs.

So I decided to start from storage alone. In order to match VMs as close as I can I gave VMWare and VBox exactly the same number of cores, ram, forced VT-x use in VMWare like in VBox and made those VMs as close to each other as I could. They also used the same .vmdk images. And results turned out to be surprisingly "okay":

So real world backup read form 8 bay btrfs RAID6 with 8 cores in VMs:

bare metal: 780 - 800 MB/s
VMWare: 700-750 MB/s
VBox: 640 - 710 MB/s

Values are rough because single megabytes don't matter and I didn't bother calculating precise average. But anyways the point is - it's acutally... really decent. I mean okay there are some performance differences but still average 680 comparing to bare metal 780 is really decent. I'd be totally okay with such performance.

However real issues are introduced by latency penalties in VBox networking - as I mentioned VirtualBox in multi-core config suffers from terrible latency up to (ouch) 8 - 10 ms ! (yes, miliseconds, not nanoseconds). Such massive latency drops NFS from in-guest 680 MB/s to (sic!) 23 MB/s. It's really really bad. VMWare is not perfect either but keeping latency below 0.3 ms allowed it to keep up with stable 130 - 140 MB/s over NFS which is pretty bad but well... it's still more than gigabit so I'm "okay" with it I guess... Not that I have any better option anyways.

What's getting more interesting is what I said earlier - in single core mode VBox does not suffer from network latency. It's marginally higher than VMWare but almost negligible difference (below 0.4ms). Despite significantly lower computing power and i/o throughput (~250) with single core - actually after exporting over NFS it's much, much faster than multi core config - around mentioned 120 MB/s. Maybe it also benefits from jumbo frames a bit (VMWare doesn't support jumbo). So yeah - IF VirtualBox would have the same latency in multi core as in single core difference between VMWare and VBox would be almost negligible.

In order to remove old VBox version factor I performed latency tests on latest VBox in OpenSUSE repo on other machine (5.1.22) but latency issue is around the same, maybe slightly better (~5 ms) but maybe this difference is caused by plenty of other factors including hardware difference so well... It's still bad.

---

I came across topic of terrible VBox network latency multiple times. It varies from something like I experience to OHMAHGAWDSLOW unusable levels but problem definitely exists. Obvious question that it brings is - are VBox developers aware of networking issues? Multiple times I heard that VBox "is supposed to run multiple lightweight vms unlike vmware which is meant for huge, heavy VMs". In such case VBox should be able to handle well those "many small VMs". Many small VMs which are usually put into some virtualized network. I'm using virtual routers (MikroTik RouterOS in VM edition (CHR)) with virtualbox and I used many VBox VMs for malware testing and infrastructure emulation (I'm profesionally related to that topic) and actually I faced latency issues plenty of times. I'm forced to run all router VMs in single core mode because of really bad latency in multi core mode. It kind of ruins whole sense of assigning multiple cores to VM if it's destroys network performance that badly. I think it's serious problem that should be addressed if VBox wants to stay competitive with "industry standard" and it should because it's great piece of software and I just wasted like 6h trying to compile those damn kernel modules that just NEVER compile properly without patching source in VMWare and obviously require recompiling and repatching after every kernel update lol.