High IRQ load caused by network traffic

Discussions about using Linux guests in VirtualBox.
Post Reply
Steffen M.
Posts: 17
Joined: 12. Sep 2013, 16:56

High IRQ load caused by network traffic

Post by Steffen M. »

Hi all,

we are running a bunch of Linux guests (Ubuntu 20.04) on Solaris 11.4 x64 SRU50 hosts using VirtualBox 7.0.2. The guests' networks are bridged to their host, the host is connected with 10 Gigabit/s Ethernet.

Unfortunately, the guests' network throughput is really low. TCP Connections deliver at maximum about 1.3 Gbit/s. When a guest application has to send a lot of (UDP) packets per time, the IRQ load gets extraordinarily high and the compute power of one logical core (the one which handles the network card's interrupts) defines the overall throughput. It can be clearly observed that, for example, "ksoftirqd/7" consumes nearly 100% of one core before the application starts to react sluggish.

We tried a lot in order to tackle this down, for example:
  • Activated X2APIC,
  • deactivated the Spectre mitigations,
  • played with the number of vCores from 6 to 12 (the host is a NUMA architecture with 2 sockets, 16 physical and 32 logical cores),
  • changed the core affinity at the host site to one socket,
  • changed the type of the emulated network controller,
  • used "ethtool" to change the offloading properties of the guest NIC,
  • tried to balance IRQs on the guest side.
Interestingly, the "virtio-net" performs even worse than "Intel PRO/1000MT Desktop (82540EM)" or "Intel PRO/1000MT Server (82545EM)". The virtualization interface we use is KVM.

My questions: Is there any possibility to reduce the IRQ load in a high packet-rate scenario? Is there any possibility to give more than one RX queues and TX queues to the emulated NIC in order to distribute the IRQ load in a multi-core scenario? Is there any way to use bonding (also known as EtherChannel or link aggregation) via several NICs between host and guest? This might also lower the stress per core on the guest's side. Is it possible to pass-through the NUMA-awareness from host to guest?

We do see the similar behavior with several guests on several hosts. We have also encountered this with VirtualBox 6.1.x (e.g. 6.1.38) and 7.0.2, so it is definitely not a specific problem of VirtualBox 7.x.

For anyone who might be interested, I've attached the VBOX file. Thank you very much in advance for any helpful hint!

Kind regards,
Steffen
Attachments
miaplacidus.vbox.gz
VirtualBox config file of an affected guest system
(1.93 KiB) Downloaded 5 times
Post Reply