Assessing host capacity for number of running guests

[sonicbuddha]

I have a linux host set up to run several virtual guests. In assessing the host performance I've noticed a high load (1-2+) when running multiple machines that grows as I run more guests; there is very little load when running no guests. In trying to track down the possible bottleneck, find any possible aspects of the host I can upgrade, I've noted that the cpu is still 99% idle, there is very little disk I/O (as measured by dtstat), plenty of free ram, and no swapping. I'm not seeing any performance issues, on either host or guest.

Example dstat output when guests are running:

Code: Select all

-----time----- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- ---load-avg---
  date/time   |usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw | 1m   5m  15m 
07-08 20:59:23|  1   9  90   0   0   0|   0     0 | 362B 1498B|   0     0 | 215   659 | 1.5  1.7  1.6
07-08 20:59:24|  0   2  98   0   0   0|   0     0 | 446B 2612B|   0     0 | 237   894 | 1.4  1.7  1.6
07-08 20:59:25|  0   0 100   0   0   0|   0  4096B| 230B  606B|   0     0 | 210   909 | 1.4  1.7  1.6
07-08 20:59:26|  0   3  96   0   0   0|   0     0 | 380B  696B|   0     0 | 233   991 | 1.4  1.7  1.6

And a few minutes after putting them into a save state:

Code: Select all

-----time----- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- ---load-avg---
  date/time   |usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw | 1m   5m  15m 
07-08 21:05:13|  0   0 100   0   0   0|   0     0 | 264B 1400B|   0     0 | 211    20 |   0  0.7  1.2
07-08 21:05:14|  0   0 100   0   0   0|   0     0 | 132B  508B|   0     0 | 208    14 |   0  0.7  1.2
07-08 21:05:15|  0   0 100   0   0   0|   0     0 | 330B 3822B|   0     0 | 212    26 |   0  0.6  1.2

Is this something to be concerned about? Are there other aspects that I am missing that can be monitored/upgraded? Can a high load, even if it does not seem to be cpu or disk bound, cause instability?

I can provide more specific details (hardware, os, etc) if it helps.

[sonicbuddha]

Funny how rereading your own post can bring some things to the fore. What was immediately evident to me after posting the dstat output was that the only thing, besides the load, that was different between when running guests and not was the increase of 'csw', which stands for context switches, when the cpu switches from one process to another. I guess its not very surprising that, when running multiple virtual machines, that the number of context switches would go up, regardless of actual cpu usage.

Not familiar with context switches (but google is my friend), the little research that I have done suggests that this shouldn't be a problem, particularly if there is no corresponding increase of hardware interrupts (int), which doesn't seem to be the case. Nor does this seem to be as much a hardware issue as operating system. Still, I'd like a more "professional" opinion.

My host is Ubuntu 8.04 running a 64bit 2.6.24-24-server kernel:

Code: Select all

uname -a
Linux gaia 2.6.24-24-server #1 SMP Fri Jul 24 22:44:54 UTC 2009 x86_64 GNU/Linux