Does a context switch factor of 10+ guest vs. host indicate some problem already?

[ams_tschoening]

I have a problem with some workloads in one of my VMs, which result in very high CPU load and an almost unusable VM anymore: SSH connections are established very slowly, character input in the terminal is slow, using the cursor keys in apps like midnight commander is slow etc. When the problem occurs, the physical host context switches per second are getting pretty high, easily up to 125'000, 175'000 or even some spikes with 250'000. Before changing to KVM and reconfiguring the VM to use default settings, things got even worse with context switches of 500'000 and more per second. One thing I noticed it that the high number of context switches in the host is not directly reflected in the guest:

That physical host is hosting 3 VMs, where 2 of those are more or less idle all the time and only host some special OpenVPN server. Both VMs have context switches of ~25 per second. The 3rd VM hosts a lot of services like web servers, databases and other things, so is constantly doing at least a little bit, resulting in context switches of ~500 per second. Most of the time the assigned CPUs are idle, though. The load on the physical host is pretty low as well, but the context switches are ~5000 per second and seem pretty high to me. I would have expected them to be around the sum of what the VMs have, maybe with some additional overhead factor, but far less then 10.

If I put some problematic workload in the 3rd VM, which is e.g. restarting ClamAV daemon, I can see that the context switches in the host increase, while keeping almost the same in the VM. It's ~500-600 in the VM vs. ~10'000-20'000 and 40'000-50'000. While all the time ClamAV is eating 100% CPU and takes minutes instead of expected seconds to restart. If additional workload gets applied using the web services or such, context switches rise even more and the VM is starting to get unusable.

The host has 12 CPU cores with Hyperthreading enabled, so 24 logical cores. 12 cores are assigned to the VMs using 1/1/10. The VM with the problematic VM had only 6 cores assigned before, but the problem was exactly the same. Therefore I'm not convinced yet that some overcommit of CPUs is the problem here. Especially because another physical host with the same hardware and more VMs all consuming 12 cores in theory as well doesn't show the same problem when restarting ClamAV, even though it has ~7'500 per second by default already. ClamAV restarted in a VM on that host doesn't increase context switches that dramatically over a long period of time, but to only ~10'000 for few seconds and that's it.

So, what is the expected context switch rate of all VMs vs. physical host? Is 5'000 or 7'500 per second for a system with very little load in htop etc. a sign for some problem already? Can I somehow assign the context switches in the host to VMs or some completely unrelated things?

Thanks!

[mpack]

A "world switch" (context switching a core between host and guest) is an expensive operation, so the ideal is to do it 0 times per second, by dedicating cores to the VM and host.

The host has 12 CPU cores with Hyperthreading enabled, so 24 logical cores.

There is no such thing as a "logical core". "Core" refers to the piece of silicon which implements what would in the past have been a complete CPU. Take 12 CPUs from the noughties (ignore whether or not they support multiple pipelines), tile them onto a single piece of silicon, add L3 cacheing of the external data bus and wrap the whole lot in appropriate BGA packaging. Now you have a 12 core CPU.

A high penalty for context switches, plus slugging down of the host, is precisely the penalty you pay for ignoring the true resources available.

A thread is a pipeline management feature, it does not involve extra silicon, it is not a core (logical or otherwise). The cost of running both threads at once kind of depends on whether they clash, e.g. both want access to their shared FPU or bus at the same time (if it's two copies of one thread then presumably they must clash).

If you have a bunch of CPU bound tasks and still want the OS's to be responsive then you'd leave say 2 cores for the host, and give 2 cores each to (up to) 5 VMs. I'd expect that to provide reasonable performance from each VM.

Of course, if the processes are I/O bound rather than CPU bound then the distinction between cores and threads is irrelevant, because that isn't where the bottleneck is. An "OpenVPN server" sounds like it's mainly doing I/O?

[ams_tschoening]

A "world switch" (context switching a core between host and guest) is an expensive operation, so the ideal is to do it 0 times per second, by dedicating cores to the VM and host.

So in your opinion the high number of context switches in the physical host is already a sign for overcommitting the CPU, even though the VMs are idle? Or are you saying that I need to manually pin/dedicate processes to cores because else for some reason VirtualBox or the Linux scheduler is context switching/moving tasks for no reason? In the end, "someone" needs to be responsible for the context switches in the host and I'm wandering if it's really the VMS and why, if they are idle and have very few context switches themselfs etc.

There is no such thing as a "logical core".

Operating systems and lots of other people out there use that term, so do I if it's appropriate. So your point is disabling Hyperthreading and the physical host will do less context switches? All benchmarks I know of don't show any problem with Hyperthreading and VM performance, instead VMs always seem to benefit if all resources of the host are be used:

https://unix.stackexchange.com/question ... ber-of-phy
http://envobi.com/post/virtualbox-hyper ... -surprise/

Additionally, I ran a 1/1/6 setup before with the same problem and in that setup I didn't even use all physical cores available.

A high penalty for context switches, plus slugging down of the host, is precisely the penalty you pay for ignoring the true resources available.

But my VM is not really creating such a high load exhausting all resources, it doesn't execute e.g. 10 CPU or I/O heavy tasks, and the host is not slugging down, it's mainly the VM itself. Restarting ClamAV itself consumes one vCPU only out of 10 and increases the context switches in the host already massively by a factor of 10 or more. So during idle I have ~5000 context switches, restarting ClamAV increases that easily to 50'000, by still only consuming one vCPU in the VM. I don't see how that is overcommitting a CPU or exhausting the host's resources.

Some pretty old discussions about CPU scheduling for VirtualBox and VMWare indicate that it might be possible that a VM can only run if ALL assigned vCPUs can be scheduled in the physical host, even if only one of it would be actually required. So it that still/really the case?

https://www.reddit.com/r/linux/comments ... es_guests/

If you have a bunch of CPU bound tasks and still want the OS's to be responsive then you'd leave say 2 cores for the host, and give 2 cores each to (up to) 5 VMs. I'd expect that to provide reasonable performance from each VM.

So one single VM with a workload that requires more than 2 CPUs in theory is already too big in your opinion and should be splitted? Even though most of the time it's still not consuming the assigned vCPUs?

Of course, if the processes are I/O bound rather than CPU bound then the distinction between cores and threads is irrelevant, because that isn't where the bottleneck is. An "OpenVPN server" sounds like it's mainly doing I/O?

The OpenVPN-VMs are mainly idle with very few clients only pinging around once a minute and those clients are IoT devices providing ~50 kB of data once a day.

[ams_tschoening]

A high penalty for context switches, plus slugging down of the host, is precisely the penalty you pay for ignoring the true resources available.

I did another test using "stress --cpu 10" in the VM and htop is showing all vCPUs are under load, but the VM itself keeps responding. I don't see any of the performance problems when I e.g. restart ClamAV. During that test, the host has context switches of around 70'000-80'000 per second, which is what ClamAV itself is sometimes generating with only one core under load. The host keeps responding as well, no performance problems I'm aware of. The context switches in the VM are around 500, which makes sense to me because the "stress" instances are user mode instances only. htop only shows large green columns as well, which is different with e.g. ClamAV. htop in the host instead shows large red columns only, so all vCPUs map to kernel mode threads.

This is another hint that CPU overcommitting is not the problem to me.