15 to 18 Machine Limit on Version 5 and above

[klaus]

Oh, and since something is fishy with signals (and/or process crashes), it is worth having a look at the output of the "dmesg" command. It could give helpful hints what's crashing when, i.e. if it actually is a VBoxSVC crash. On many linux distros the output of this kernel log also ends up in /var/log/messages or some other log file, which would give longer term data on the situation.

[FitchT]

I get "Qt Warning: QThread::start: Thread creation error: Resource temporarily unavailable" then it just sits there.

[klaus]

That warning is extremely scary, and shows the same problem as some of the logs before... the question is which resources are exhausted. To me it looks like a seriously misconfigured/miscompiled kernel, or you're really exceeding some resource limits.

[Martin]

Maybe something similar to this?
https://unix.stackexchange.com/question ... 4-3-kernel

What is the output of 'ulimit -a' on your system?

[FitchT]

Yes when that last VM is started the GUI crashes, shows all the VM's as powered off, but you can RDP to them and use as normal.
I've been trying some more different combinations of headless and normal etc to try any find a new variable but no success thus far.
The short answer I've been running on since April is to downgrade to 4.3 any server with 15 of more VMs.
I'd like to test using the 7 RH kernel but haven't had time just yet and the 22 servers I have now are all 6.9, as you know there is no upgrade path to 7, only rebuild.

[FitchT]

I've been trying many combinations to find a new variable with no success. Also the result of starting the last VM is now not consistent. Twice now, it has not crashed the GUI but did stop new SSH sessions under the same user (script) it was running as. You can still ssh in as root.
I'm attaching a ps aux of before and after

[FitchT]

ulimit shows

Code: Select all

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 516091
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 516091
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[FitchT]

top shows:

Code: Select all

top - 09:30:31 up 19:55,  1 user,  load average: 0.18, 0.35, 0.81
Tasks: 435 total,   1 running, 433 sleeping,   0 stopped,   1 zombie
Cpu(s):  1.2%us,  6.6%sy,  0.0%ni, 90.4%id,  1.7%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  132285400k total, 118254392k used, 14031008k free,   177744k buffers
Swap:  4194300k total,     5576k used,  4188724k free, 30007928k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23902 script    20   0 6579m 4.1g 4.1g S 37.1  3.3   3:33.95 VirtualBox
23409 script    20   0 6593m 4.1g 4.1g S 13.6  3.3   5:44.17 VirtualBox
22825 script    20   0 10.8g 8.3g 8.2g S 10.6  6.6  10:59.99 VirtualBox
24550 script    20   0 5915m 3.6g 3.6g S  7.0  2.9   2:09.79 VirtualBox
23297 script    20   0 6585m 4.1g 4.1g S  6.3  3.3   6:10.18 VirtualBox
23184 script    20   0 6585m 4.1g 4.1g S  5.6  3.3   6:14.11 VirtualBox
23352 script    20   0 6583m 4.1g 4.1g S  5.6  3.3   6:15.43 VirtualBox
24258 script    20   0 6583m 4.1g 4.1g S  5.6  3.3   3:11.66 VirtualBox
24369 script    20   0 6577m 4.1g 4.1g S  5.6  3.3   4:02.30 VirtualBox
22939 script    20   0 6585m 4.1g 4.1g S  5.3  3.3   5:41.49 VirtualBox
22995 script    20   0 6524m 4.1g 4.1g S  5.3  3.3   6:01.94 VirtualBox
23054 script    20   0 6577m 4.1g 4.1g S  5.3  3.3   6:08.27 VirtualBox
23241 script    20   0 6581m 4.1g 4.1g S  5.3  3.3   6:11.01 VirtualBox
23957 script    20   0 6575m 4.1g 4.1g S  5.3  3.3   4:17.02 VirtualBox
24012 script    20   0 6579m 4.1g 4.1g S  5.3  3.3   4:17.60 VirtualBox
22884 script    20   0 6388m 4.0g 4.0g S  5.0  3.2   6:11.52 VirtualBox
24067 script    20   0 6577m 4.1g 4.1g S  5.0  3.3   4:13.26 VirtualBox
24313 script    20   0 6577m 4.1g 4.1g S  5.0  3.3   3:50.38 VirtualBox
24122 script    20   0 6587m 4.1g 4.1g S  4.6  3.3   4:18.37 VirtualBox
22751 script    20   0 2198m  16m 7180 S  2.7  0.0   1:38.64 VBoxSVC
22746 script    20   0  196m 3708 2052 S  1.3  0.0   0:56.63 VBoxXPCOMIPCD
24850 root      20   0 15296 1528  948 R  0.7  0.0   0:00.18 top
22472 script    20   0  356m 4096 2752 S  0.3  0.0   0:12.97 pulseaudio
22713 script    20   0 1249m  51m  30m S  0.3  0.0   0:39.08 VirtualBox
    1 root      20   0 19340 1372 1092 S  0.0  0.0   0:02.49 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.11 kthreadd
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.03 migration/0
    4 root      20   0     0    0    0 S  0.0  0.0   0:03.37 ksoftirqd/0
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/0
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.05 watchdog/0
    7 root      RT   0     0    0    0 S  0.0  0.0   0:00.23 migration/1
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/1

[FitchT]

Dmesg output

[michaln]

"[Hardware Error]: Machine check events logged" <-- that is Not Good. Indicates failing hardware.

[FitchT]

I hear you. Keep in mind please that I have 22 servers running a few hundred VMs with the exact same issue. I can reproduce at will on any of them.
They are all CentOS (Redhat) 6.9, all but one are Dell Poweredge, mixed SAS, SATA, 1 CPU 2 CPU and a variety of RAID cards and ram amounts.

[FitchT]

I've just installed 5.2.2 same issue, no difference.

[michaln]

I've just installed 5.2.2 same issue, no difference.

That's entirely expected -- there's no fix because we don't have the slightest idea what the problem could possibly be, and you seem to be the only one having it...

[FitchT]

I hear you. Unfortunately, we'll just have to live with it as we have and slowly move to vmware. I hate to do it, VBox has been fantastic for us, but we're out of answers and the environment is continously growing.

[Perryg]

So VMware is going to allow you to have "22 Redhat 6.9 servers running several hundred machines" for free? Go figure.