virtualbox.org

I have a Ubuntu Server running 14.04 VirtualBox 5.0.16r105871 which is also running an appliance called M3VMREP2 which is a mysql replication server. During peek transactions the appliance stops at a state of "guru meditation". This is not an isolated event and we have been able to replicate this on 3 other servers running similar configurations. We must have the appliance miss-configured. I have included the"showvminfo" at the time of the crash and also the Vbox.log file.
Any assistance would be appreciated.



Supporting Information:
==============
Server Dell R815 / 64 Cores and 512G memory
root@A105:/home/appliance# free
total used free shared buffers cached
Mem: 528366096 425850100 102515996 10908 548436 197508600
-/+ buffers/cache: 227793064 300573032
Swap: 5324796 1728816 3595980
-----
root@A105:/home/appliance# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/A105--vg-root 9.1G 3.7G 4.9G 44% /
/dev/mapper/arm-app 1008G 342G 616G 36% /data
/dev/mapper/A105--vg-home 120G 60M 114G 1% /backup2
/dev/sdc1 17T 4.2T 12T 27% /home
-----
appliance@A105:/home/dsanders$ VBoxManage --version
5.0.16r105871
==============

Virtual Appliance Configuration
Ubuntu 14.04.4 LTS / 32 Cores / 128G Memory
----
dsanders@M3VMREP2:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
-------
dsanders@M3VMREP2:~$ free
total used free shared buffers cached
Mem: 132035852 19449960 112585892 5908 286720 6931056
-/+ buffers/cache: 12232184 119803668
Swap: 2093052 0 2093052
----
dsanders@M3VMREP2:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/RMS--VM--Server--1--vg-root 7.6G 2.4G 4.8G 34% /
/dev/sda1 236M 40M 184M 18% /boot
/dev/sdb 20G 5.9G 13G 32% /home
/dev/sdc 296G 99G 182G 36% /data

===============
-- VM Info at Time of Guru
appliance@A105:~$ VBoxManage showvminfo M3VMREP2v5
Name: M3VMREP2v5
Groups: /
Guest OS: Ubuntu (64-bit)
UUID: fb9775f9-7d5b-46bd-95ec-f29450054d06
Config file: /home/appliance/VirtualBox VMs/M3VMREP2v5/M3VMREP2v5.vbox
Snapshot folder: /home/appliance/VirtualBox VMs/M3VMREP2v5/Snapshots
Log folder: /home/appliance/VirtualBox VMs/M3VMREP2v5/Logs
Hardware UUID: fb9775f9-7d5b-46bd-95ec-f29450054d06
Memory size: 131072MB
Page Fusion: off
VRAM size: 12MB
CPU exec cap: 100%
HPET: off
Chipset: piix3
Firmware: BIOS
Number of CPUs: 32
PAE: off
Long Mode: on
CPUID Portability Level: 0
CPUID overrides: None
Boot menu mode: message and menu
Boot Device (1): Floppy
Boot Device (2): DVD
Boot Device (3): HardDisk
Boot Device (4): Not Assigned
ACPI: on
IOAPIC: on
Time offset: 0ms
RTC: UTC
Hardw. virt.ext: on
Nested Paging: on
Large Pages: off
VT-x VPID: on
VT-x unr. exec.: on
Paravirt. Provider: Default
State: guru meditation (since 2016-03-17T11:52:21.119000000)
Monitor count: 1
3D Acceleration: off
2D Video Acceleration: off
Teleporter Enabled: off
Teleporter Port: 0
Teleporter Address:
Teleporter Password:
Tracing Enabled: off
Allow Tracing to Access VM: off
Tracing Configuration:
Autostart Enabled: off
Autostart Delay: 0
Default Frontend:
Storage Controller Name (0): SATA
Storage Controller Type (0): IntelAhci
Storage Controller Instance Number (0): 0
Storage Controller Max Port Count (0): 30
Storage Controller Port Count (0): 4
Storage Controller Bootable (0): on
Storage Controller Name (1): IDE
Storage Controller Type (1): PIIX4
Storage Controller Instance Number (1): 0
Storage Controller Max Port Count (1): 2
Storage Controller Port Count (1): 2
Storage Controller Bootable (1): on
SATA (0, 0): /home/appliance/VirtualBox VMs/M3VMREP2v5/M3VMREP2v5-disk1-os.vdi (UUID: 65f0a84f-2604-49d7-8e0c-a7f496de99d9)
SATA (1, 0): /home/appliance/VirtualBox VMs/M3VMREP2v5/M3VMREP2v5-disk2-home.vdi (UUID: 9c980d1c-4446-4ae1-91e4-44e9859e1d3b)
SATA (2, 0): /home/appliance/VirtualBox VMs/M3VMREP2v5/M3VMREP2v5-disk3-data.vdi (UUID: 4db3bd66-1814-49b0-839d-d93dc1765d67)
SATA (3, 0): /home/appliance/VirtualBox VMs/M3VMREP2v5/M3VMREP2v5-disk4_extra.vdi (UUID: 8fbc596e-c5fc-4b08-a63e-75302589b248)
IDE (1, 0): Empty
NIC 1: MAC: 080027186761, Attachment: Bridged Interface 'em1', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth group: none
NIC 2: disabled
NIC 3: disabled
NIC 4: disabled
NIC 5: disabled
NIC 6: disabled
NIC 7: disabled
NIC 8: disabled
Pointing Device: PS/2 Mouse
Keyboard Device: PS/2 Keyboard
UART 1: disabled
UART 2: disabled
UART 3: disabled
UART 4: disabled
LPT 1: disabled
LPT 2: disabled
Audio: disabled
Clipboard Mode: disabled
Drag and drop Mode: disabled
Session name: headless
Video mode: 640x480x32 at 0,0 enabled
VRDE: enabled (Address 0.0.0.0, Ports 9000-9100, MultiConn: on, ReuseSingleConn: off, Authentication type: null)
VRDE port: 9000
Video redirection: disabled
VRDE property: TCP/Ports = "9000-9100"
VRDE property: TCP/Address =
VRDE property: VideoChannel/Enabled =
VRDE property: VideoChannel/Quality =
VRDE property: VideoChannel/DownscaleProtection =
VRDE property: Client/DisableDisplay =
VRDE property: Client/DisableInput =
VRDE property: Client/DisableAudio =
VRDE property: Client/DisableUSB =
VRDE property: Client/DisableClipboard =
VRDE property: Client/DisableUpstreamAudio =
VRDE property: Client/DisableRDPDR =
VRDE property: H3DRedirect/Enabled =
VRDE property: Security/Method =
VRDE property: Security/ServerCertificate =
VRDE property: Security/ServerPrivateKey =
VRDE property: Security/CACertificate =
VRDE property: Audio/RateCorrectionMode =
VRDE property: Audio/LogPath =
USB: disabled
EHCI: disabled
XHCI: disabled
USB Device Filters:
Available remote USB devices:
Currently Attached USB Devices:
Bandwidth groups:
Shared folders:
VRDE Connection: not active
Clients so far: 1
Last started: 2016/03/16 13:41:58 UTC
Last ended: 2016/03/17 11:52:23 UTC
Sent: 0 Bytes
Average speed: 0 B/s
Sent total: 0 Bytes
Received: 0 Bytes
Speed: 0 B/s
Received total: 0 Bytes
Video capturing: not active
Capture screens: 0
Capture file: /home/appliance/VirtualBox VMs/M3VMREP2v5/M3VMREP2v5.webm
Capture dimensions: 1024x768
Capture rate: 512 kbps
Capture FPS: 25
Guest:
Configured memory balloon size: 0 MB
OS type: Linux26_64
Additions run level: 2
Additions version: 4.3.28 r100309
Guest Facilities:
Facility "VirtualBox Base Driver": active/running (last update: 2016/03/16 13:43:10 UTC)
Facility "VirtualBox System Service": active/running (last update: 2016/03/16 13:43:46 UTC)
Facility "Seamless Mode": not active (last update: 2016/03/16 13:43:10 UTC)
Facility "Graphics Mode": not active (last update: 2016/03/16 13:43:10 UTC)

The mismatch of versions can cause an issue. You should update your extension pack You are assigning all CPU cores to the guest. This can cause issues because there is an overhead for each core you add and the host will get starved out causing hangs and other errors. Lower it until you see if the guest gets stable. ( I see you stated 64 cores but remember that AMD calls threads cores but they are not. Each core has 2 threads but that is not the same as a real core ) You have plenty of memory and if the guest is the main reason for this server I would increase the RAM more. You may also be hitting a ceiling in the host and need to adjust for that. The low memory indicates the host is reporting it low IIRC and that could also be because of the over-commitment of your processor IMHO. Scan the syslogs of the server and see if you can find a reason as well. Something that is out of the ordinary. From the looks of it you are swapping a lot and that does not make a lot of sense. Is this host used for something other than running this guest?

Perryg, Thanks for the quite response.

I agree that the Oracle VM VirtualBox Extension Pack is out of date and needs to be updated. I will do that.

I will also lower the number of cores from 32 to 24 for the appliance if you think that will help the stability.

You asked regarding the Physical Server, its sole purpose is to run VM's, it currently has 4 other smaller VM's running on it which also experience the same guru meditation issue. I did not want to cloud the issue with other details unless you asked, which you did. You said "From the looks of it you are swapping a lot". I looked at the server and it is not showing that I am swapping at all? are you speaking of the VM and if so how can I further identify the cause?

Let me know if you have any other suggestions as I make the changes you identified to this VM.

My mistake on the swap, I slipped on on the "0"

If all the other guest are doing the same then it would indicate a starved server. Make sure you do not provide more memory or processor than the host has available. Meaning that you must calculate the total used by all guests and not exceed the amount that is available. These resources are not shareable. Also you need to leave some for the host side to work properly or you will have lags, hangs, and even lock ups.

I have updated the Guest Additions so now all VM's on the server are at 5.0.16r105871.

I do not have an issue with memory, if I add all the allocated memory for all the VMs its less then 1/2 if the total memory.

The problem is with vCPUs. Yes I have AMDs and I have a total of 64 cores as identified by htop and top. You mentioned that the VMs have a different allocation then "one to one" for the AMD processors. I am not familiar enough with hardware and cores to opine. However, if I sum the total VMs CPU allocation, I am exceeding the total number of available cores. However, they will never all be used at the same time due to the separation of the applications. Example VM-A will perform Task 1 and when done, VM-B will perform Task 2 and when done VM-C will perform Task 3, etc. If this is the case, even tough the sum of all the CPUs exceed the servers total the process prevents it from ever happening. Is there an issue with this sort of configuration or implementation that prevents this in VirturalBox? I guess the real question is, can I have a 4 CPU server and have 5 VMs each allocating 1-core? The symptom I am experiencing is the crashing of the VMs in a "Guru Meditation" condition, which I have not found any technical description which would identify what this means, the cause, and/or remedy.

Let me know what you think and thanks again for all your time.

I guess I didn't make myself clear. You do not have 64 true cores. You have 32 cores and hyper threading. This is not new and there are several lawsuits against AMD for hyping the number of true cores to the level of threads. Look at the code snippets I posted from your log file to see what is truly going on.

Anyway I have gone as far as I can go without actually touching the unit. If you can show any other reason for this I don't know what more to tell you.

Thanks for all the help. I will continue to test the VM using the changes suggested. If I have anymore issues I will create a new tread.