FreeBSD network speed

[Perryg]

Just to point out, the guest additions have nothing what-so-ever to do with networking. So they will not be an issue in that regards.

[bobloblian]

going in circles, have read everything I can find twice now, no more ideas are coming to me. So... can I make an unusable system even worse? Turns out I can, but I figured out how to bring it back...

my goal was to try virtio, just to see. For this, I needed to upgrade the box to 8.2. not the most painful process, but really appreciating apt now. figured out how to make the virtio drivers load, good for some testing. no apparent difference in tests whether the guest has one or two cpus, each test run 3 times.

guest=>server: 33-35 MB/s: 80%cpu guest: 2cpu@50% host
host=>server: 94-96 MB/s: 2cpu@20% host
guest=>host: 32.5-36MB/s: 80%cpu guest: 2cpu@50% host
host=>guest: 28MB/s: 100%cpu guest: 2cpu@60% host

So, an improvement, although a fairly underwhelming one.

FWIW, I decided to try the virtualbox 5.0 Beta package. no noticeable difference in performance. perhaps in the next day or two I will try some older versions, see if that makes any difference...

[noteirak]

Puzzled here too. Just to be sure, I tried from my host to my guest with 1GB zero file. The guest is connected via host-only using Intel/Pro 1000MT Desktop interface
I reach 68.3MB/s with one CPU@100% (2 configured in the VM) and one CPU@88% on the host.
Looks to me more like a processing limit, possibly coupled with a not-so-good driver in BSD?

[bobloblian]

That's interesting, thanks for sharing that info. That leads me to agree with your conclusion that the combination of virtualization and OS contains a performance ceiling and it has been found.

I am pretty sure the custom app will break at some point by doing so, but I think the next step is to try to keep upgrading the guest BSD. If I can get the speed up to 50MB/s, it will still be slow in some cases but might be good enough for overall production use...

[Perryg]

I suspect the approx 60MB/sec has more to do with the drive speed than it does with the network speed.

Try moving a 10GB file between two different drives on metal and see what the transfer speed really is. Like I said I work with large files all the time.

[noteirak]

it was actually from a RAM drive to another RAM drive

[Perryg]

Really. Well if you really want to test the network speed you need to use iperf and see what it says.

Code: Select all

mint@mint-17 ~ $ iperf -c 192.168.1.10
------------------------------------------------------------
Client connecting to 192.168.1.10, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.110 port 44647 connected with 192.168.1.10 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.5 GBytes  9.03 Gbits/sec

I still would love to know what your drive to drive speed is on metal.

[noteirak]

I'm fairly sure the speed is limited by the encryption/decryption here. It will always make transfers slower, and the speeds we see so far do not really surprise me.
iperf will show you pure network speed, but is not adapted here. The OP would like more speed, but I am not sure it is even possible in his configuration/hardware.

Can't give you the drive to drive speed, I have lots of caching in place for the running VMs, so it wouldn't be realistic. Since this is production, I won't mess with the system

[Perryg]

Fair enough. Yes translation is a factor and drive speed and format overhead.

Take for instance you have a drive in ext4 and a drive in ntfs. The translation caused even on metal will reduce the overall transfer speed do by at least 10% to 20%.
Even the chipset and RAM has a factor in it. I can get bursts of approx 300MB/sec on file sizes between 1GB and 5GB but it drops down to approx 170 GB/sec over 5GB. This is drive to different drive on same host ( metal ) with x99 chipset that can support that speed. older chipsets I usually got around 60MB to 100MB but never more.

So the way I see it when using file transfer to judge the speed of the network is not that realistic and there a way more things in play and it always falls to the weakest link (drive speed)
Add the translation of it being virtual and you realistically should see around 60MB/sec or less, but I really think it has more to do with everything else beside network speed IMHO. Oh and don't get me started on green drives. I wish I had never purchased one. Its great for power saving but actual transfer speed suffers greatly. I only have one left and use it for backup and storage now.

[noteirak]

I really think it has more to do with everything else beside network speed IMHO

I think we both try to say the same thing from the start! To me, OP speeds are still in the "expected" range given the scenario.

[bobloblian]

Pulling data from the metal BSD box across the network is about 75 MB/s, though in production context, a windows machine usually pulls a file at around 60 MB/s. The application context is a CAD environment, where it is not unusual to have files in the 1-2 GB range, or to be pulling a whole bunch of 20-50MB files.

In my case, the guest/host system is running on a RAID array of 8 drives, and if I did my math correctly should support a write throughput of around 1100Mbit/s. The only protocol overhead would be smb, so by the theory, drive speed should not be the bottleneck on a gigabit network. Especially not to the degree I am observing in the guest, and by contrast not observing on the host.

It did occur to me a few days ago that drive speed could be the factor, so I tried to find BSD tools for measuring that. there doesn't seem to be a lot, apparently a program called diskinfo or such is the best you can get, so I ran it and compared it to the metal server. There is an outside, middle and inside transfer rate. The middle and inside were not significantly different, but the outside is an order of magnitude lower on the host/guest than it is on the metal server. I dismissed that before, but perhaps this deserves more investigation.

+1 on the remark about green drives. Call me an energy hog, but I dont' even like them for storing backups...

[bobloblian]

so nothing worked that I tried. Took a step back from the problem and started over.

I put all files on the host, and using a combination of mount_smbfs so the application on the guest has access to the files, and msdfs to redirect the windows workstations to the much faster host, I think I have hit a system that will function with sufficient speed to satisfy production needs. Was still a bit complicated to get everything mapped out correctly, but by comparison turned out to be a very simple and elegant solution...

Many thanks and cases of karmic beers for your help...