15 to 18 Machine Limit on Version 5 and above

Discussions related to using VirtualBox on Linux hosts.

Re: 15 to 18 Machine Limit on Version 5 and above

Postby klaus » 20. Nov 2017, 18:24

Oh, and since something is fishy with signals (and/or process crashes), it is worth having a look at the output of the "dmesg" command. It could give helpful hints what's crashing when, i.e. if it actually is a VBoxSVC crash. On many linux distros the output of this kernel log also ends up in /var/log/messages or some other log file, which would give longer term data on the situation.
klaus
Oracle Corporation
 
Posts: 534
Joined: 10. May 2007, 14:57

Re: 15 to 18 Machine Limit on Version 5 and above

Postby FitchT » 20. Nov 2017, 20:01

I get "Qt Warning: QThread::start: Thread creation error: Resource temporarily unavailable" then it just sits there.
FitchT
 
Posts: 17
Joined: 15. Nov 2017, 23:18

Re: 15 to 18 Machine Limit on Version 5 and above

Postby klaus » 20. Nov 2017, 20:12

That warning is extremely scary, and shows the same problem as some of the logs before... the question is which resources are exhausted. To me it looks like a seriously misconfigured/miscompiled kernel, or you're really exceeding some resource limits.
klaus
Oracle Corporation
 
Posts: 534
Joined: 10. May 2007, 14:57

Re: 15 to 18 Machine Limit on Version 5 and above

Postby Martin » 20. Nov 2017, 20:24

Maybe something similar to this?
https://unix.stackexchange.com/question ... 4-3-kernel

What is the output of 'ulimit -a' on your system?
Martin
Volunteer
 
Posts: 1822
Joined: 30. May 2007, 18:05
Primary OS: Fedora 18
VBox Version: PUEL
Guest OSses: XP, Win7, Linux, OS/2

Re: 15 to 18 Machine Limit on Version 5 and above

Postby FitchT » 21. Nov 2017, 15:46

Yes when that last VM is started the GUI crashes, shows all the VM's as powered off, but you can RDP to them and use as normal.
I've been trying some more different combinations of headless and normal etc to try any find a new variable but no success thus far.
The short answer I've been running on since April is to downgrade to 4.3 any server with 15 of more VMs.
I'd like to test using the 7 RH kernel but haven't had time just yet and the 22 servers I have now are all 6.9, as you know there is no upgrade path to 7, only rebuild.
FitchT
 
Posts: 17
Joined: 15. Nov 2017, 23:18

Re: 15 to 18 Machine Limit on Version 5 and above

Postby FitchT » 21. Nov 2017, 16:28

I've been trying many combinations to find a new variable with no success. Also the result of starting the last VM is now not consistent. Twice now, it has not crashed the GUI but did stop new SSH sessions under the same user (script) it was running as. You can still ssh in as root.
I'm attaching a ps aux of before and after
Attachments
processlistafterstartinglastvm.txt
(37.8 KiB) Not downloaded yet
processlistbeforestartinglastvm.txt
(37.69 KiB) Not downloaded yet
FitchT
 
Posts: 17
Joined: 15. Nov 2017, 23:18

Re: 15 to 18 Machine Limit on Version 5 and above

Postby FitchT » 21. Nov 2017, 16:30

ulimit shows
Code: Select all   Expand viewCollapse view
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 516091
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 516091
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
FitchT
 
Posts: 17
Joined: 15. Nov 2017, 23:18

Re: 15 to 18 Machine Limit on Version 5 and above

Postby FitchT » 21. Nov 2017, 16:31

top shows:
Code: Select all   Expand viewCollapse view
top - 09:30:31 up 19:55,  1 user,  load average: 0.18, 0.35, 0.81
Tasks: 435 total,   1 running, 433 sleeping,   0 stopped,   1 zombie
Cpu(s):  1.2%us,  6.6%sy,  0.0%ni, 90.4%id,  1.7%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  132285400k total, 118254392k used, 14031008k free,   177744k buffers
Swap:  4194300k total,     5576k used,  4188724k free, 30007928k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23902 script    20   0 6579m 4.1g 4.1g S 37.1  3.3   3:33.95 VirtualBox
23409 script    20   0 6593m 4.1g 4.1g S 13.6  3.3   5:44.17 VirtualBox
22825 script    20   0 10.8g 8.3g 8.2g S 10.6  6.6  10:59.99 VirtualBox
24550 script    20   0 5915m 3.6g 3.6g S  7.0  2.9   2:09.79 VirtualBox
23297 script    20   0 6585m 4.1g 4.1g S  6.3  3.3   6:10.18 VirtualBox
23184 script    20   0 6585m 4.1g 4.1g S  5.6  3.3   6:14.11 VirtualBox
23352 script    20   0 6583m 4.1g 4.1g S  5.6  3.3   6:15.43 VirtualBox
24258 script    20   0 6583m 4.1g 4.1g S  5.6  3.3   3:11.66 VirtualBox
24369 script    20   0 6577m 4.1g 4.1g S  5.6  3.3   4:02.30 VirtualBox
22939 script    20   0 6585m 4.1g 4.1g S  5.3  3.3   5:41.49 VirtualBox
22995 script    20   0 6524m 4.1g 4.1g S  5.3  3.3   6:01.94 VirtualBox
23054 script    20   0 6577m 4.1g 4.1g S  5.3  3.3   6:08.27 VirtualBox
23241 script    20   0 6581m 4.1g 4.1g S  5.3  3.3   6:11.01 VirtualBox
23957 script    20   0 6575m 4.1g 4.1g S  5.3  3.3   4:17.02 VirtualBox
24012 script    20   0 6579m 4.1g 4.1g S  5.3  3.3   4:17.60 VirtualBox
22884 script    20   0 6388m 4.0g 4.0g S  5.0  3.2   6:11.52 VirtualBox
24067 script    20   0 6577m 4.1g 4.1g S  5.0  3.3   4:13.26 VirtualBox
24313 script    20   0 6577m 4.1g 4.1g S  5.0  3.3   3:50.38 VirtualBox
24122 script    20   0 6587m 4.1g 4.1g S  4.6  3.3   4:18.37 VirtualBox
22751 script    20   0 2198m  16m 7180 S  2.7  0.0   1:38.64 VBoxSVC
22746 script    20   0  196m 3708 2052 S  1.3  0.0   0:56.63 VBoxXPCOMIPCD
24850 root      20   0 15296 1528  948 R  0.7  0.0   0:00.18 top
22472 script    20   0  356m 4096 2752 S  0.3  0.0   0:12.97 pulseaudio
22713 script    20   0 1249m  51m  30m S  0.3  0.0   0:39.08 VirtualBox
    1 root      20   0 19340 1372 1092 S  0.0  0.0   0:02.49 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.11 kthreadd
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.03 migration/0
    4 root      20   0     0    0    0 S  0.0  0.0   0:03.37 ksoftirqd/0
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/0
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.05 watchdog/0
    7 root      RT   0     0    0    0 S  0.0  0.0   0:00.23 migration/1
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/1
FitchT
 
Posts: 17
Joined: 15. Nov 2017, 23:18

Re: 15 to 18 Machine Limit on Version 5 and above

Postby FitchT » 21. Nov 2017, 16:33

Dmesg output
Attachments
dmesg.txt
(56.33 KiB) Downloaded 5 times
FitchT
 
Posts: 17
Joined: 15. Nov 2017, 23:18

Re: 15 to 18 Machine Limit on Version 5 and above

Postby michaln » 21. Nov 2017, 17:20

"[Hardware Error]: Machine check events logged" <-- that is Not Good. Indicates failing hardware.
michaln
Oracle Corporation
 
Posts: 2841
Joined: 19. Dec 2007, 15:45
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Any and all

Re: 15 to 18 Machine Limit on Version 5 and above

Postby FitchT » 21. Nov 2017, 18:35

I hear you. Keep in mind please that I have 22 servers running a few hundred VMs with the exact same issue. I can reproduce at will on any of them.
They are all CentOS (Redhat) 6.9, all but one are Dell Poweredge, mixed SAS, SATA, 1 CPU 2 CPU and a variety of RAID cards and ram amounts.
FitchT
 
Posts: 17
Joined: 15. Nov 2017, 23:18

Re: 15 to 18 Machine Limit on Version 5 and above

Postby FitchT » 27. Nov 2017, 23:13

I've just installed 5.2.2 same issue, no difference.
FitchT
 
Posts: 17
Joined: 15. Nov 2017, 23:18

Re: 15 to 18 Machine Limit on Version 5 and above

Postby michaln » 28. Nov 2017, 09:42

FitchT wrote:I've just installed 5.2.2 same issue, no difference.

That's entirely expected -- there's no fix because we don't have the slightest idea what the problem could possibly be, and you seem to be the only one having it...
michaln
Oracle Corporation
 
Posts: 2841
Joined: 19. Dec 2007, 15:45
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Any and all

Re: 15 to 18 Machine Limit on Version 5 and above

Postby FitchT » 28. Nov 2017, 16:15

I hear you. Unfortunately, we'll just have to live with it as we have and slowly move to vmware. I hate to do it, VBox has been fantastic for us, but we're out of answers and the environment is continously growing.
FitchT
 
Posts: 17
Joined: 15. Nov 2017, 23:18

Re: 15 to 18 Machine Limit on Version 5 and above

Postby Perryg » 28. Nov 2017, 16:28

So VMware is going to allow you to have "22 Redhat 6.9 servers running several hundred machines" for free? Go figure.
Perryg
Site Moderator
 
Posts: 34265
Joined: 6. Sep 2008, 22:55
Primary OS: Linux other
VBox Version: OSE self-compiled
Guest OSses: *NIX

PreviousNext

Return to VirtualBox on Linux Hosts

Who is online

Users browsing this forum: Bing [Bot] and 19 guests