rcapping VB Machine memory usage

[junkfer]

Hello Gurus, Engineers, Magicians, Everyone....

Need a BIG help from someone.... I have a Solaris 10 update 9 x86 OS installed on a Sun x2270 2xQuad Core CPU 6 GB RAM. (EIS patched with version 2010.12.) BIOS and SP patches are on the top level. VirtualBox version is 3.2.12.

The server have 2 functions:
- File serving as webserver from a whole-root zone with dedicated 2 core cpu-pool and rcap-ed memory limit 1,5GB, in the zone i have Iplanet 7.0.9 + NSAPI PHP webserver running in project rcapped its memory limit to 768MB
- Running VirtualBox Windows 2003Server VM for streaming media content to the internet. (running as root user in VBoxHeadless mode, from project, with dedicated 2 core cpu-pool) Not heavy loaded server, used only 3 days of the week, this times the guest usage is 1,5 hour long, 50%CPU load, 25MB/sec network load,

The functions working well, almost everything is fine. I think I have no problem with the whole server... BUT
I have rebooted the server today after a few days of configuration experiences to check the whole picture.
So currently i have the following results:
- Today at 12:00AM reboot
- Today at 12:00AM continous "vmstat 5" sad that i have 2,5GB free memory
- Yet at 19:00PM (7. hour uptime) same "vmstat 5" say that i have 900MB (!!!) free memory; "prstat -J" (or "rcapstat" too) sad that the zone's webserver use 161MB RAM.... AND i see in this same "prstat" that the VB project use only 48MB (RSS) RAM, and the VirtualBox VBoxHeadless process using 2166MB (RSS) Memory

At the last days i tried 2 different way to use the VirtualBox VBoxHeadless from project (rcap.max-rss=sizeinbytes and project.pool="poolname")
- as normal non-root user with project user."username" i tried run the guest that i log in with this user and run "Vboxheadles -startvm "vmname" &"
- as a project named "VBPROJ" i started the guest "newtask -p VBPROJ Vboxheadles -startvm vmname &"
eighter way i experienced in "prstat -J" that the project or the user."username" uses only 50MB (RSS) RAM + the process use its configured 2GB (RSS) RAM + "rcapstat" shows, that the project use only 50MB.....

So my question that how could i the VirtualBox rcapped to one rcap-jail, or how could i the VB memory usage controll and follow???

Before the last weekend i had the problem with the server, that after cca. 3-4 day running "freezed" and logged nothing to nowhere, i could only RESET the server from the SystemProcessor. This was the reason to put the zone and within the webserver in this double rcap-jail. If it is possible i want do so way with the VB guest too.....

Could someone help, or i open a ticket for it?
Many thanks, for any help, with best regards,

Ferenc Junkert

[Ramshankar]

VirtualBox allocates memory for VMs via the kernel and therefore zones are irrelevant. Hence, there is no way to explicitly cap memory. If you have a growing memory situation please open a bug in our Public Bugtracker with all the relevant details and VM logs.

You've talked about the memory cap you have assigned your zone(s), which has no real effect, but you've not mentioned how much memory each of your VMs have been assigned. We can explore options like RAM preallocation but first we need to understand your configuration accurately.

[junkfer]

Hello,

Thanks the answer, and the help... So the configuration:

Sun X2270 with 2 pcs. 2GHZ Xeon 4core; 6 GB RAM; 2x500GB HDD
OS.: Solaris 10 u9 (EIS patchset 2010.12. is installed)
FS: UFS mixed with ZFS slice (ZFS ARC size is limited to 805MB)
CPU pools: 1x2cores for the zone, 1x2cores for the virtualbox, 1x4core for the "global"
Zone: 1pcs. Whole root zone, own cpu pool, own RCAPPED memory 1,5GB, 1 own ZFS dataset, zone.max-swap 1GB
The zone running one Oracle Iplanet webserver in a project which rcap memory limit is 768MB
VB: Version 3.2.12, running currently from the "global" AND from project, the project has its own CPU pool.
Memory size 2GB, PAE and Nested paging is enabled. Running guest type is Windows 2003Server.

The server was booted tomorrow at 3 PM, at this time i have the following resources:
vmstat 5 3 | tail -1 >
0 0 0 7570680 2224252 0 10 2 0 0 0 0 0 0 0 0 2100 965 1146 0 1 99

currently
0 0 0 7484500 2130644 0 9 0 0 0 0 0 0 0 0 0 2082 1011 1173 0 1 99

typical echo "::memstat"| mdb -k

Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 100801 393 6%
ZFS File Data 187811 733 12%
Anon 136337 532 9%
Exec and libs 15459 60 1%
Page cache 12612 49 1%
Free (cachelist) 31082 121 2%
Free (freelist) 1084388 4235 69%

Total 1568490 6126
Physical 1553653 6068

The "nice" thing is, that the VB machine has no load, must nothing to do, only run, currently... I have every day less memory with cca 100MB.

Before the last day i had the problem, that the server freezed after every 3-4 days run (REBOOT: RESET from ILoM). That was the reason while i capped the zfs arcsize. Before that point, the time under i have lost almost the whole memory was 1 day. (Memory loss percentage in that case was 90%)


Actually i have a Dell PE1800 Server too, with 450days uptime, the only differences, that its running with VB 2.1.4 BUT the VB configuration is "almost" same without any reboot. Okay it is running on Solaris 10 u6, and has only to do the VB service to run, but THIS server is running.... I could not understand while this most better server is freezing. Cold we do some tests?


Thanks for the help..... regards,


Ferenc Junkert

[junkfer]

Hello,

Ramshankar, maybe i found something. Actually the following in the VB Manual,

Warning: Do not run other hypervisors (open-source or commercial virtualization
products) together with VirtualBox!

so it is a problem, that i have a running zone (what is actually a virtualization product) paralell with the VB?

Could be this is the reason of my memory loss, or the "global" system freezes?

Thanks,

Junkfer

[junkfer]

Hi all,

So.... i had time at this weekend and reconfigured the server... And the result is.... i dont know what i mean....

The new configuration is that i moved the VirtualBox service into a zone (the VB Guest configuration not changed!!!)... I added cpu pool and capped memory to the zone, but seems th VirtualBox HAS so power that he allocated memory direct from KERNEL....

vmstat 5 3
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m0 m1 m2 m1 in sy cs us sy id
0 0 0 8156268 3065888 401 1333 271 1 41 0 1394 25 16 16 17 2612 14597 5090 1 4 95
0 0 0 7732352 2431888 0 12 0 0 0 0 0 0 0 0 0 2554 1005 1236 0 1 99
0 0 0 7732236 2431740 0 2 0 0 0 0 0 0 0 0 0 2617 1085 1284 0 1 99

echo "::memstat"|mdb -k
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 643548 2513 41%
ZFS File Data 145343 567 9%
Anon 104527 408 7%
Exec and libs 18448 72 1%
Page cache 10658 41 1%
Free (cachelist) 25870 101 2%
Free (freelist) 620096 2422 40%

Total 1568490 6126
Physical 1553653 6068

rcapstat -z
id zone nproc vm rss cap at avgat pg avgpg
1 VBoxZone - 74M 76M 3072M 0K 0K 0K 0K
2 MessagesZone - 448M 331M 1536M 0K 0K 0K 0K

The zfs_cache is limited at 805MB.

Ramshankar,
- how i would have enough memory if the usage of the server are begin growing??
(Make/Could I lower the value of the RCAP ?mcap physcap? for the VBox zone?)
- In this case i dont understand what is the pourpose/advantage that i run "in a zone the VB"??

Tomorrow this time i would have more info about the memory situation.

Thanks for any help or rtfm document.... regards,

Ferenc Junkert

[Ramshankar]

Hello,

Ramshankar, maybe i found something. Actually the following in the VB Manual,

Warning: Do not run other hypervisors (open-source or commercial virtualization
products) together with VirtualBox!

so it is a problem, that i have a running zone (what is actually a virtualization product) paralell with the VB?

Could be this is the reason of my memory loss, or the "global" system freezes?

Thanks,

Junkfer

No, Solaris zones is not a Hypervisor that alters the CPU VT-x/AMD-V states etc. Therefore this should have nothing to do with memory loss or "global" system freezes you seem to mention.

[nxnlvz]

I have noticed an issue for a while now that will seem to cause a global system freeze. If a guest, Debian Linux in this case, has a very high memory load and starts to swap it may cause a system freeze. It is not clear to me when this started to happen but it is now consistent.

In this particular case the VM it is running Zimbra 7.x. and is the only active VM on this Solaris Express host. Zimbra seems to have a memory leak that will cause its usage to grow day after day. That is a separate issue. The result is that I can predict almost to the hour when it will consume all free memory and start to swap. At 2 GB that is about 8 days. At 3 GB that is about 15 days. As long I would restart the guest before this time the host was fine. Because of this I also increased the allocated memory to the guest to extend the uptime / time before restarting the guest. Of course the solution was imperfect as all solutions of this type are and the freeze came a day sooner than expected this morning.

The host host system memory is running at only 50% utilization. ZFS is capped. The guest is assigned only one CPU core.

I have not been able to catch any log data up to this point since the system is completely dead so all data is anecdotal. No core from VB is present. Any suggestions for capturing data on this would be appreciated. It is not a vital system at this time so it can stand being offline for a few hours. The system freezes occur at about 3am when the daily cleanup occurs.

[Ramshankar]

Which version of VBox are you trying? The original poster mentions 3.2.12.

Have you capped the ZFS max. ARC sizes?

[nxnlvz]

4.1.x Versions now. No more 3.x on my systems.

The ZFS ARC sizes have been capped but that brings up a good point. If there is contention for memory and ZFS does not give it up that may cause the issue. The heavy usage on guest VM will be after a period of realtive idle which in theory would allow for some parts of the guest to be paged out. I will lower the cap on ZFS to the bare minimum and see what happens.

Alongside this, is there a way to lock the entire VBox guest in memory so that it is never paged or swapped out?

-nxn

[Ramshankar]

VirtualBox does NOT page out the guest memory. It's locked down as it gets allocated. The ZFS ARC size requirement is imposed due to the way VirtualBox allocates kernel pages, which is unknown to ZFS and therefore ZFS cannot really relinquish it's reserved cache memory because it has no knowledge of VirtualBox's requests, on the other hand, VirtualBox cannot use the standard way of allocating memory because it has special requirements (including alignment, boundary, contiguity, physically/virtual & combinations thereof.)

What you -can- do is preallocate the entire guest memory, before launching the VM run:

VBoxManage setextradata <vm_name> VBoxInternal/RamPreAlloc 1

This would force VirtualBox to preallocate & lock down the entire guest RAM instead of doing it on a demand (aka lazy alloc).

[nxnlvz]

That last bit to set RamPreAlloc is useful information.

A little update on this. Since the last huge update to Solaris 11 the problem seems to have gone away. I have no doubt that there were myriad changes that would have affected the operation of VBox but there definitely changes to the allocator is seems.

Thanks!