Patch for increasing maximum guest RAM size

[HubTou]

Hello,

Here's a little patch I made in order to increase the maximum guest RAM size on a 64 bits host (from 16 GB to 48 GB in the attached example).

As I do not yet have the hardware to test this and the machine I plan to use is quite expensive, I'll be glad if someone could test this before I commit myself. Launching and successfully using a VM with more than 16 GB would be enough for me at the moment.

Please note that under some operating systems, there might be additional actions to be taken in order to increase the maximum process RAM size.
For example, on FreeBSD-amd64 (under which I use VirtualBox), kern.maxdsiz should be set in /boot/loader.conf if more than 32 GB is needed: with kern.maxdsiz=51539607552 in my case.

Thanks in advance!

Best regards,

Hubert

[frank]

This patch will not work as it is not enough to raise the limit but the VMM must also support it. The current guest limit is 16GB. This limit will be raised in future versions but this will require much more changes and tests.

[HubTou]

Hello Franck

You must be talking about the Virtual Memory Manager, instead of the Virtual Machine Monitor (the VMM in VirtualBox source code)?

If yes, my understanding is that this is mostly/entirely an operating system issue. If the underlying OS support such process memory usage, it should be possible to run VMs with more than 16 GB. Of course, this feature would be OS dependent.

Do you envision additional parts of VirtualBox source code that would need to be modified in order for this to work?

Anyway, I will have the required hardware by next tuesday. I will report the results of my tests here. If you want to make some tests of your own, I can also provide root access for you while this is in testing?

Best regards,

Hubert

---
For reference, under 64 bits FreeBSD, the system default process limits are the following:

Code: Select all

# limits
Resource limits (current):
  datasize             33554432 kB   <---- 32 GB
  stacksize              524288 kB
  coredumpsize         infinity kB
  memoryuse            infinity kB
  memorylocked         infinity kB
  sbsize               infinity bytes
  vmemoryuse           infinity kB
  swapuse              infinity kB

The datasize may be raised, at boot time, using the tunable variable I previously mentioned.

[frank]

To raise the maximum limit of a VM process you have to modify several sources in src/VBox/VMM -- the virtual machine monitor. There are several constraints within that code, and believe me, it is not about just changing the size of an array or something trivial like that.

[HubTou]

Hello Franck,

Well, it *seems* to work nonetheless!

Here's an example with a 40 Gb guest:



Unlike with some other guest OS, the memory is really allocated on the host (and thus starting the VM is a bit slow):



Here are the VM parameters:



And the VirtualBox limits on the host:



By the way, i see there's a 2 Tb limit on the VDI files, but i've also been able to create 2.5 Tb static VDI files. Is it just a "safety" limit for OS not designed to handle bigger file systems or should i expect troubles ahead?

Best regards,

Hubert

PS: I've casted a quick glance at the VMM sources but found nothing obvious. I will continue my tests with a huge business intelligence application.

[sandervl]

That's a premature conclusion. Wait until the guest actually uses that much RAM and things will blow up.

[HubTou]

Hello,

That's a premature conclusion. Wait until the guest actually uses that much RAM and things will blow up.

I try to use words carefully. I don't think being conclusive by saying:

it *seems* to work
[...]
I will continue my tests

Anyway, speaking about tests, what about this one: allocating a larger than 16 Gb RAM disk in the guest operating system and writing every sectors inside:


Please note the options for mdconfig and newfs:

From mdconfig(8) manual page:

Code: Select all

 -o [no]option
	     Set or reset options.
	     [no]reserve
		     Allocate and reserve all needed storage from the start,
		     rather than as needed.

From newfs(8) manual page:

Code: Select all

  -E      Erase the content of the disk before making the filesystem.  The
	     reserved area in front of the superblock (for bootcode) will not
	     be erased.
	     NB: Erasing may take as long time as writing every sector on the
	     disk.

I was a bit surprised at the memory size displayed in the guest OS, but it explained itself in the VirtualBox logs for this machine:

Code: Select all

# cd Machines/test/Logs
# grep -i "ram" VBox.log | head -3
00:00:00.045 Host RAM: 49084MB RAM, available: 47262MB
00:00:00.155   RamHoleSize     <integer> = 0x0000000020000000 (536870912)
00:00:00.155   RamSize         <integer> = 0x0000000a00000000 (42949672960)
# expr 42949672960 + 536870912
43486543872

I don't draw conclusions at this time because i want to make further tests with a real application, but *so far*, it still *seems* to work

If you have additional testing ideas or things to look at, please let me know and i will gladly run them for you.

Best regards,

Hubert

[Technologov]

Good news: Well, I finally managed to get a good hardware with 24 Cores and 48 GB of RAM, but limited time on it. Runs RHEL 5.

So I may try this patch, as time permits.

However, what is the procedure to *fill* such amounts of RAM ? (on Windows 64-bit guests it should be easy - MS Paint with huge bitmap resolution, but what about 32-bit PAE guests? - having a few dozens of Paints running is a bad option...)

Another big problem is: how to compile things on RHEL 5? (it doesn't have Qt4.4.x, and other libs may be lacking...)

[HubTou]

Hi Technologov!

I finally managed to get a good hardware with 24 Cores and 48 GB of RAM [...] what is the procedure to *fill* such amounts of RAM ?

Launching a VM with as much RAM as you can! Some guest OS, like FreeBSD, allocate all the memory given by the host right from their start.

A generic method to fill the guest memory would then be to build a ramdisk (like i did in the examples above), make a thorough format and stuff it with as many copies of your file systems as needed. If you want to put some load in Unices, you could also use the virtual terminals to launch several parallel kernel and userland compilations

The syntax for building ramdisks is different between OS.

Another way, would be to use something like Memtest86+ (http://www.memtest.org/#downiso) in a guest to test memory (just boot from the virtual CD without harddisk), using 32 bits + PAE or 64 bits settings.

what about 32-bit PAE guests?

I don't know, i haven't tried that, but see above.

Another big problem is: how to compile things on RHEL 5? (it doesn't have Qt4.4.x, and other libs may be lacking...)

Install the development tools (gcc & all)from the DVD? Recompiling everything needed or fetching pre-built packages somewhere?

If you don't mind thrashing your Red Hat installation and your machine has Internet access, I can provide a disk image of a ready-to-run FreeBSD 8 amd64 / VirtualBox system, to paste on your server?

Best regards,

Hubert

[Technologov]

I may thrash it, as it not yet entered production... but I prefer not to. I prefer to keep it.

>Install the development tools (gcc & all)from the DVD? Recompiling everything needed or fetching pre-built packages somewhere?

Development tools exist, but not Qt-4.4.x.

About memory tests:
* Yes, ramdisk is a good test idea, for those OSes that have it. (Windows does not)
* memtest86+ is not very good idea, because it has it's own kernel, it does not use Linux/Windows kernel. We bypass guest OS this way.
It may be used as supplemental test, but not as primary test.

[HubTou]

Hello Technologov,

Development tools exist, but not Qt-4.4.x.

Have a look there:
Building Qt 4.4 for RHEL 5 (http://solidsmoke.blogspot.com/2008/12/ ... hel-5.html)

* Yes, ramdisk is a good test idea, for those OSes that have it. (Windows does not)
* memtest86+ is not very good idea, because it has it's own kernel, it does not use Linux/Windows kernel. We bypass guest OS this way.
It may be used as supplemental test, but not as primary test.

There are plenty of ramdisks for Windows (http://www.mydigitallife.info/2007/05/2 ... 03-server/), including ones provided by Microsoft themselves (http://support.microsoft.com/?scid=kb%3 ... &x=17&y=11).

For memtest86+ (which you can declare as Linux Red Hat 64 bits in VirtualBox), you can try it as a way to verify that a guest with more than 16 GB of RAM doesn't easily blow out.

Best regards,

Hubert

[Technologov]

Lacking packages on RHEL5:
1. Qt-4.4.x (RHEL has Qt3.x)
2. curl 7.16.1 (RHEL has 7.15.x) -- I have manually compiled from sources, and installed.
3. libpulse, but I don't use it anyway, so I disabled this. I prefer ALSA.

Your way of Qt4.4.x is too difficult for me.

I have found another way: (but it doesn't work)
Qt 4.6.3 and qt-creator 1.3.1-1 updates for Centos 5.5:
http://joseph.freivald.com/linux/2010/0 ... entos-5-5/

The packages were installed (qt4 and qt4-devel, 4.6.3), but VirtualBox configure script can't detect it.... sad.
I also tried the official Qt4.6.3 SDK from Nokia, but VBox does not detect that either. I giveup on Qt for now.

I will try to build the SDL FE.

---------------
I was able to compile with SDL. VBox SVN - r30690.

Unfortunately I can't start VMs... bad. Those are new VMs with all default parameters.
# LD_LIBRARY_PATH=. ./VBoxManage startvm my32G-02 --type headless
Oracle VM VirtualBox Command Line Management Interface Version 3.2.51_OSE
(C) 2005-2010 Oracle Corporation
All rights reserved.

Waiting for the VM to power on...
ERROR:
Details: code NS_OK (0x0), component , interface , callee
#
# LD_LIBRARY_PATH=. ./VBoxSDL --startvm my32G-02
Error: RTR3Init failed rcRC=-1908

Any ideas why VMs don't start?
---------------
To sum it up -- can you provide me with binary VirtualBox for RHEL5 ? (including Qt GUI and your patches)
BTW - GUI needs to be patched, too, it seems.

I can work with SDL FE, but it is too time consuming... it takes 2x more time than with Qt4 GUI.
---------------

[Technologov]

*** RESULTS ***
Continuously updated...
I work with SDL FE, because I am unable to compile the Qt4 one.
Host: Dual AMD Opteron Magny-cours (2x12) 24 core + 48 GB of RAM + RHEL5 x64 + VBox 3.2.6 OSE with HubTou patches.

Good News: Some OSes were able to boot and work with 36 GB of RAM.
-RHEL-5.4/32-bit (PAE)
-RHEL-5.4/64-bit
-Windows Server 2003/x64
-Windows 7/x64

TODO:
A. test Guest OSes:
32-bit PAE Windows Server

Guest OSes than someone else should test:
-Mac OS X Server
-FreeBSD (HubTou -- time donor ?)
-Solaris

B. test load/RAM usage.
I have tried memtest86 v1.70 (from openSUSE 10.3)-- it fails with 8 GB assigned to VM. works fine with 4 GB. That is -- the application stucks after several seconds if VM has 8 GB of RAM.
This is not related to your patches however, as it happens with VirtualBox 3.2.6 PUEL, too.
[update] memtest86 1.65 from RHEL 5.4 works fine !
memtest86 1.65 - PASSED ! (36 GB of RAM)

[HubTou]

Hello Technologov,

First of all, thank you for taking time to test this!

Your way of Qt4.4.x is too difficult for me [...] I will try to build the SDL FE [...] To sum it up -- can you provide me with binary VirtualBox for RHEL5 ? (including Qt GUI and your patches) [...] BTW - GUI needs to be patched, too, it seems.

There's another (simpler) way: compile without Qt, without X11, with just the HeadLess client and VNC.
This is what i always do (it also explains why i haven't offered patches for the GUI), using my own vboxhost wrapper to easily change the number of CPUs and quantity of RAM.

And sorry, i can't provide RHEL5 binaries because i don't use this OS.

Good News: Some OSes were able to boot and work with 36 GB of RAM.

I successfully ran the following ones with 42 GB:
-RHEL-5.4/x64
-FreeBSD-8.0/x64 (you have the screen captures above)

You can download a fresh image of the last one (to test as i did above) at:
http://www.projet-hev.org/dist/FreeBSD- ... d64.vdi.7z
(root password is blank. BTW, this image was automatically generated by a script using the keyboardputscancode feature of VirtualBox )

Guest OSes than someone else should test: -Mac OS X Server -Solaris

I'm not sure i will have time to do this, but i might be able to test OpenSolaris 2009.06 and perform additional tests with FreeBSD 32 bits + PAE.

Best regards,

Hubert

[Technologov]

Host: Dual AMD Opteron Magny-cours (2x12) 24 core + 48 GB of RAM + VBox 3.2.6 OSE with HubTou patches.

Good news:
RHEL-5.4-32-bit (PAE) was tested under RAM load.
36GB RAM in guest. 30 GiB ramdisk (tmpfs), and is filled by 29 GiB zero-image. Works !

I have provided VBox Log + Screenshot for reference.