Page 1 of 1

Slow network speeds to/from virtual hosts

Posted: 14. Jan 2010, 11:42
by bass
Hello!

* I recently switched to opensolars as base system, in order to get ZFS and to virtualize.
* I run debian Linux on one of the virtual machines.

The problem is that network transfers to/from the virtual machines are slow.

Scenario 1, Win7 machine <-> opensolaris via SMB
both directions, I get around 75-85 megabytes / sec over my gigabit network. I am fully satisfied with this.

Scenario 2, Win7 machine <-> debian (which is a VM running under vbox), SMB and FTP
now I´m down to around 10-15 megs / sec.

Scenario 3, debian (VM) <-> opensolaris host, SMB
also around 10-15 megs /sec

So, it seems like the virtual network "magic" between the solaris host, and the VM´s, slows down the network/samba/ftp speeds really bad.
Have I confed something wrong, or is this the way it is?

My solaris machine:
AMD ATHLON II X4 620 2.6GHZ
Intel PRO/1000 nic
4gb ram
AMD-V seems to be working

Any help would be really appreciated!

Re: Slow network speeds to/from virtual hosts

Posted: 14. Jan 2010, 13:48
by TeaDaze
How have you configured the network in VirtualBox for the Debian guest?
Check in network / advanced settings to ensure you are using the Paravirtualized Network (virtio-net) otherwise VirtualBox will be emulating the hardware.

If you change the Adaptor Type you might need to fiddle with /etc/udev/rules.d/70-persistent-net-rules after booting the debian guest.

Hope that helps

Re: Slow network speeds to/from virtual hosts

Posted: 15. Jan 2010, 16:55
by dagar
I have a Centos 5.4 x86_64 guest. What changes to /etc/udev/rules.d/70-persistent-net-rules need to be made to get virtio-net to work? I have not gotten it to recognize the virtio interface.
Here is what I have:
ACTION=="add", SUBSYSTEM=="net", IMPORT{program}="/lib/udev/rename_device"
SUBSYSTEM=="net", RUN+="/etc/sysconfig/network-scripts/net.hotplug"

I have tried to readd the VB additions and shutdown the network and then run kudzu again and restart the network and still no go.

Re: Slow network speeds to/from virtual hosts

Posted: 15. Jan 2010, 17:52
by bass
TeaDaze wrote:How have you configured the network in VirtualBox for the Debian guest?
Check in network / advanced settings to ensure you are using the Paravirtualized Network (virtio-net) otherwise VirtualBox will be emulating the hardware.

If you change the Adaptor Type you might need to fiddle with /etc/udev/rules.d/70-persistent-net-rules after booting the debian guest.

Hope that helps
Hello! Tanks for the input!
I changed to virtio-net, and it helped, got around 30 megs/sec istead...
But still not close to what I get to the host OS :(

Do I need to manually install some drivers in my debian install?

Re: Slow network speeds to/from virtual hosts

Posted: 15. Jan 2010, 18:34
by TeaDaze
dagar wrote:I have a Centos 5.4 x86_64 guest. What changes to /etc/udev/rules.d/70-persistent-net-rules need to be made to get virtio-net to work? I have not gotten it to recognize the virtio interface.
Here is what I have:
ACTION=="add", SUBSYSTEM=="net", IMPORT{program}="/lib/udev/rename_device"
SUBSYSTEM=="net", RUN+="/etc/sysconfig/network-scripts/net.hotplug"

I have tried to readd the VB additions and shutdown the network and then run kudzu again and restart the network and still no go.
Newer Debian / ubuntu seem to have the virtio-net drivers available from a standard install, I assume you'll need to add those to your system if it cannot locate the devices.

The change I have to do with debian/ubuntu when changing networks (usually when moving a physical server to a virtual) is remove the network interface lines from 70-persistent-net-rules and reboot (if the driver or mac address changes it leaves the old entries and adds the new hardware. This causes what should be eth0 to move to eth1 etc. Removing them and rebooting means only the new hardware is added).
Again I don't use centos so unsure what they put in there.

Re: Slow network speeds to/from virtual hosts

Posted: 15. Jan 2010, 18:47
by TeaDaze
bass wrote:Hello! Tanks for the input!
I changed to virtio-net, and it helped, got around 30 megs/sec istead...
But still not close to what I get to the host OS :(

Do I need to manually install some drivers in my debian install?
I don't think you need to install extra drivers. It seems debian/ubuntu ship with virtio-net compatible drivers as standard now.
Are you using jumbo packets or anything like that on the host? Does the guest have similar setup?

I've not really hammered the network on any of my virtuals (we only have 100meg uplink) so I'm not sure what else to suggest.

Re: Slow network speeds to/from virtual hosts

Posted: 16. Jan 2010, 16:16
by kebabbert
MS has reworked Windows as of late, so Vista and Win7 is able to saturate Gigabit NIC out-of-the-box, whereas Linux is more trickier to do. I dont know how to make Linux saturate gigabit NIC, or if it is even possible

Re: Slow network speeds to/from virtual hosts

Posted: 18. Jan 2010, 10:58
by bass
It´s not a problem with linux/debian.
Before the switch to my current machine + solaris + virtualbox I ran my debian install on a celereon ~1,6ghz, 512 meg ram, and an onboard gigabit nic, and it could do 70-80 megs / sec (~650mbit) over the network without any problems at all.

The thing is that local writes do the "harddrive" from guest, inside virtualbox is also slow.
#:hdparm -t /dev/hda
/dev/hda:
Timing buffered disk reads: 58 MB in 3.42 seconds = 16.94 MB/sec


And if I use the shared folder function in virtualbox, it´s also as slow.

And, NFS/SMB is slow too. (between guest OS and host)


So, the problem I have to solve is:
How the ¤#¤#"¤#¤"# do I write data from inside the GUEST os, out to the host OS (solaris + zfs) drives, with full speed?


The bottleneck clearly seems to be VirtualBox/emulation(?). As soon as data have to pass in/out from a guest OS (that is, "thru" virtualbox), I loose performance BIIIG time...

Re: Slow network speeds to/from virtual hosts

Posted: 28. Jan 2010, 12:28
by marcodes
Hi bass,
you problem is not a your problem: it's a VirtualBox problem.

VirtualBox perform only 150~200 Mbits/sec on network interface for each VM. I think it's by design.
If you switch to virtio-net you can grow performance on 300~350 Mbit/sec.
If you use (f.e.) 4 VMs, you will notice that host physical machine will perform on 150*4=600 Mbit/sec but always 150 per VM. So i think it's by design or a runtime issue.
I made all this test in may days and i used iperf to exclude any hard-disk performance consideration.
Iperf is my preferred network performance tester because it'svery precise and available on both windows/Linux.

To confirm this, i only say you that if you migrate the same VMs on VMware ESX/ESXi (not VMware Server) or Citrix XenServer, the network performances will splash up on full 1 Gbit/sec!
You can try if you don't believe me! :)

Please Sun (Oracle), you should resolve this slow network performance issue.

Re: Slow network speeds to/from virtual hosts

Posted: 25. Oct 2010, 15:46
by oh_noes
bass wrote:Scenario 1, Win7 machine <-> opensolaris via SMB
both directions, I get around 75-85 megabytes / sec over my gigabit network. I am fully satisfied with this.

Scenario 2, Win7 machine <-> debian (which is a VM running under vbox), SMB and FTP
now I´m down to around 10-15 megs / sec.

Scenario 3, debian (VM) <-> opensolaris host, SMB
also around 10-15 megs /sec


I have the EXACT same problem. I'm alos running the exact same VM's. Host OS Solaris with Saba runs really fast, 80MB/s network read/writes to/from Windows 7. However, when I trey to use the Guest VM (Debian) it's slow and I only get 10-15MB/s.

hdparm -t /dev/sda on the Guest OS Debian returns fast disk (100MB+).

I'm currently using 82545EM to get the 10-15MB/s via Samba to the Debian guest. If I change this to virtio, it drops to not even 1MB/s. So something is definitely broken here. I couldnt find anything in 70-net-persistence (or whatever the real file is) that hints to virtio. lsmod listed virtio_net loaded though.

VBoxHeadless and Solaris 10 U9 and ZFS Mirror. Intel Quad Core, 8GB Memory (2GB to Debian). ZFS ARC is capped at 4GB. Any ideas?

Any ideas?

Re: Slow network speeds to/from virtual hosts

Posted: 25. Oct 2010, 15:51
by marcodes
we should wait Oracle solution :?

Re: Slow network speeds to/from virtual hosts

Posted: 27. Oct 2010, 21:19
by binaryMercenary
I realize this clip below was for the KVM, but I thought I'd share my findings as many Virtualizers have compenents like these in common, or can be tested in a similar fashion. I post this here because the comments on this page were helpful to me and I've got some related content below too.

Anyway, please enjoy and try for yourself in order to determine what the best combo is for you.
Also remember you may have to trade speed for bandwidth....

Three machines present as follows:
10.0.4.52 running fc13 host with kvm 0.12, br0 off of eth0
10.0.4.222 running fc10 as guest os, varying but runs lighttpd/lighty with the server-status page enabled for those in the subnet....
10.0.4.53 running fc13 with kvm 0.12
***Note that the e1000 driver does NOT provide these gains if host enviro is fc12!!!

In order to test and find the most ideal (for our combination)
Shutdown machine (not just reboot). Change driver.
Then Run test cases against each driver
case1: favorable short ping like a marketer would advertise.
case2: longer ping, 40-50 times to allow for fair snapshot
case3: ping whilst doing ab attack as follows:
ab -n 15000 -c 20 -k http://10.0.4.222/server-status & ping 10.0.4.222
Note the the ab attach is peformed from a fourth machine in the same subnet with apache/lighty...
But also note that this isn't the important benchmark over the ping 1 and 2, and the iperf
Results:
case4 - running iperf -s on server and iperf -c 10.0.4.222
e1000 1rtt min/avg/max/mdev = 0.154/0.196/0.244/0.026 ms
e1000 2rtt min/avg/max/mdev = 0.133/0.188/0.247/0.031 ms
e1000 3rtt min/avg/max/mdev = 0.246/7.568/102.257/25.346 ms
e1000 [ 3] 0.0-10.0 sec 811 MBytes 680 Mbits/sec
virtio 1rtt min/avg/max/mdev = 0.393/0.455/0.595/0.049 ms
virtio 2rtt min/avg/max/mdev = 0.394/0.447/0.558/0.039 ms
virtio 3rtt min/avg/max/mdev = 0.486/0.744/1.403/0.334 ms
virtio [ 3] 0.0-10.0 sec 1.06 GBytes 908 Mbits/sec
pcnet 1rtt min/avg/max/mdev = 0.262/0.316/0.490/0.050 ms
pcnet 2rtt min/avg/max/mdev = 0.236/0.824/7.877/1.683 ms
pcnet 3rtt min/avg/max/mdev = 0.299/3.136/10.048/3.027 ms
pcnet [ 3] 0.0-10.0 sec 447 MBytes 374 Mbits/sec
rtl8139 1rtt min/avg/max/mdev = 0.188/0.214/0.252/0.021 ms
rtl8139 2rtt min/avg/max/mdev = 0.162/0.219/0.324/0.043 ms
rtl8139 3rtt min/avg/max/mdev = 0.347/3.099/7.139/2.312 ms
rtl8139 [ 3] 0.0-10.2 sec 15.2 MBytes 12.5 Mbits/sec
hypvDef 1rtt min/avg/max/mdev = 0.183/0.202/0.220/0.014 ms
hypvDef 2rtt min/avg/max/mdev = 0.164/0.257/3.019/0.376 ms
hypvDef 3rtt min/avg/max/mdev = 0.292/3.162/6.884/2.775 ms
hypvDef [ 3] 0.0-10.3 sec 15.1 MBytes 12.3 Mbits/sec
ne2kpci 1rtt min/avg/max/mdev = 0.365/0.383/0.416/0.030 ms
ne2kpci 2rtt min/avg/max/mdev = 0.312/11.844/69.885/25.413 ms
ne2kpci 3rtt min/avg/max/mdev = 0.401/50.870/206.575/72.708 ms
ne2kpci NA: <1MB??? STALLED. NOTE ping min is poor, avg is worse!


Some other abandonned thoughts on this:

Network options, in the host environment, such as bridge...
Bridged networking
Network Address Translation (NAT)
VDE (Virtual Distributed Ethernet) networking
Internal networking
Host Alias??? Sure, you could ugly-hack a ton of the faster but lower bw interfaces...
Host-only networking
Not attached
TUN works with IP frames
TAP works with Ethernet frames. NOTE TUN is lower to kernel...
PPP vs?

A file that has at least some relationship to my favorite new e1000e driver...
/etc/udev/rules.d/70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.
# Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Serdes) (rule written by anaconda)
# Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Serdes) (rule written by anaconda)
# PCI device 0x8086:0x1098 (e1000e) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:30:48:94:53:6b", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
# PCI device 0x8086:0x1098 (e1000e) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:30:48:94:53:6a", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

Re: Slow network speeds to/from virtual hosts

Posted: 22. Nov 2010, 13:54
by oh_noes
binaryMercenary, everything you just said doesn't make any sense.