VERR_NO_LOW_MEMORY error

[zdinu]

Hello,

I am trying to fit many 2GB Linux VMs with no guestmemoryballoon into a 192GB OpenIndiana host. I have limited the max zfs arc size to 10GB and therefore I have about 180 GB free for the VMs. I have currently running 49 VMs, from which two of them with 4GB and the rest of 47 with 2GB memory. Therefore the total memory allocated by the VMs is about 102GB. When I try to spin up the VM with number 50 which has 2GB memory configured, I got the following error:

VM creation failed (GVMM) (VERR_NO_LOW_MEMORY)}

Here is the actual Log file:

VirtualBox 4.1.18 r78361 solaris.amd64 (Jun 5 2012 16:10:17) release log
00:00:00.072 Log opened 2012-07-23T11:29:49.330377000Z
00:00:00.072 OS Product: SunOS
00:00:00.072 OS Release: 5.11
00:00:00.072 OS Version: oi_151a4
00:00:00.073 DMI Product Name: S5520HC
00:00:00.073 DMI Product Version: ....................
00:00:00.077 Host RAM: 196539MB RAM, available: 139123MB
00:00:00.077 Executable: /opt/VirtualBox/amd64/VBoxHeadless
00:00:00.077 Process ID: 8913
00:00:00.077 Package type: SOLARIS_64BITS_GENERIC
00:00:00.083 Installed Extension Packs:
00:00:00.083 Oracle VM VirtualBox Extension Pack (Version: 4.1.18 r78361; VRDE Module: VBoxVRDP)
00:00:00.096 VRDP: Statistics created: [full], enabled: 0.
00:00:00.098 VRDP: VRDP: VD: Frames=10 MinMS=15 MaxMS=300 HistoryMS=2000 VideoMS=300
00:00:00.100 VRDP: TCP server listening on port 5956.
00:00:00.101 VRDE: loaded version 4 of the server.
00:00:00.101 VRDE: [SCARD]
00:00:00.103 SUP: Opened VMMR0.r0 (/opt/VirtualBox/amd64/VMMR0.r0) at 0xfffffffff8c90020.
00:00:00.149 VMSetError: /home/vbox/tinderbox/4.1-sol-rel/src/VBox/VMM/VMMR3/VM.cpp(814) int vmR3CreateU(UVM*, uint32_t, int (*)(VM*, void*), void*); rc=VERR_NO_LOW_MEMORY
00:00:00.149 VMSetError: VM creation failed (GVMM)
00:00:00.155 ERROR [COM]: aRC=NS_ERROR_FAILURE (0x80004005) aIID={1968b7d3-e3bf-4ceb-99e0-cb7c913317bb} aComponent={Console} aText={VM creation failed (GVMM) (VERR_NO_LOW_MEMORY)}, preserve=false
00:00:00.192 Power up failed (vrc=VERR_NO_LOW_MEMORY, rc=NS_ERROR_FAILURE (0X80004005))
00:00:00.447 VRDP: TCP server closed.

Now, the questions:
1) Is there any limit for the total number of running VMs ?
2) Is there any other memory limit other then the total physical memory ?
3) What does VERR_NO_LOW_MEMORY stands for ?

One other thing to note is that I'm using 3145728 for cbHyperHeap memory for each VM, which is set with the following command:
VBoxManage setextradata <vm_name> "VBoxInternal/MM/cbHyperHeap" "3145728"

Please let me know what can I do next...

Thank you,
Dinu

[Ramshankar]

1) Is there any limit for the total number of running VMs ?

I don't remember the hard limit (or if we ever found a system large enough to hit the hard limit), I've personally tested it on servers with up to 500 VMs running fine, the actual limit is well beyond it. I'll try to update this bit when I find the time to look at it (either the code or some other reliable source)
Update: The limit is 1023 VMs on 64-bit hosts (but probably not for Windows hosts)[1] IIRC, there is the SysV IPC semaphore limit that's partly or fully responsible, I don't think the limit exists in the VMM code.

2) Is there any other memory limit other then the total physical memory ?

With VirtualBox 4.1.x the theoretical total VM memory limit is 2 TiB, (tested up to 1 TiB). The per-VM memory limit is also 2 TiB. How you split this 2 TiB between VMs is therefore irrelevant.

3) What does VERR_NO_LOW_MEMORY stands for ?

VirtualBox requires a bit of physical memory below 4 GiB for every VM you start. Once the host is exhausted of memory below this range and when VirtualBox can't gobble up the required memory, it gets sad & refuses to start the VM. On Solaris hosts, VirtualBox (up to version 4.1.16) is allocating memory in an overly aggressive manner when large pages are involved (see https://www.virtualbox.org/ticket/10600), the side effect of which could be exhausting the lower memory space, I haven't been able to confirm this with testing. If your VBox.log contains "EnableLargePages" with "1" then there are chances that 4.1.20 might fix this issue for you (when it's released).

One other thing to note is that I'm using 3145728 for cbHyperHeap memory for each VM, which is set with the following command:
VBoxManage setextradata <vm_name> "VBoxInternal/MM/cbHyperHeap" "3145728"

This is setting the hypervisor heap memory area size to 3 MiB, which is fine.

[1] Source: VirtualBox 2.1.2 changelog.