Page 1 of 1
Memory configuration
Posted: 30. Oct 2017, 19:30
by bauer40
Hi Gurus!
I wonder about a few memory settings for a server running VirtualBox and simultaniously acting as a small file server.
The server has 24GB memory, running Solaris 11.3. The sum of RAM for the running VMs is 11.25GB, and the sum of the processes rss (and vss) is close to this value. However, my kernel memory seems to be exhausted:
Code: Select all
root@vmhost:~# echo "::memstat" | mdb -k
Page Summary Pages Bytes %Tot
---------------------------- ---------------- ---------------- ----
Kernel 5601305 21.3G 89% <-------- Wow!!!!
Defdump prealloc 171182 668.6M 3%
ZFS 82094 320.6M 1%
Anon 165536 646.6M 3%
Exec and libs 5788 22.6M 0%
Page cache 94703 369.9M 2%
Free (cachelist) 239 956k 0%
Free (freelist) 168349 657.6M 3%
Total 6289196 23.9G
There is a documentation (
https://docs.oracle.com/cd/E53394_01/ht ... gnwox.html) on setting user_reserve_hint_pct to limit zfs arc cache to grow too large and throw "application" memory out. But I'm not sure if VBox allocates "normal" memory, or utilizes kernel memory.
So could somebody clarify a few points for me, please?
1) do Virtual Machines started with VBoxHeadless allocate the per-guest configured memory as kernel memory, or is it "regular" memory? (of course I know a few MB kernel memory will be used by the network drivers, I care more about those 4GB virtual RAM for a VM)
2) is this memory locked in RAM, so it can't be paged out?
3) should I limit the arc cache using the mentioned procedure to make sure my VMs are held in memory?
Thank you for your precious feedback,
Peter
Re: Memory configuration
Posted: 30. Oct 2017, 19:47
by bauer40
Addendum:
When I stop a VM rated with 4GB, the kernel memory allocation shrinks:
Code: Select all
root@infra1:~# echo "::memstat" | mdb -k
Page Summary Pages Bytes %Tot
---------------------------- ---------------- ---------------- ----
Kernel 4537441 17.3G 72% <--------------- shrunk by the size of the stopped VM
Defdump prealloc 171182 668.6M 3%
ZFS 91644 357.9M 1%
Anon 155966 609.2M 2%
Exec and libs 5821 22.7M 0%
Page cache 93864 366.6M 1%
Free (cachelist) 7365 28.7M 0%
Free (freelist) 1225913 4.6G 19%
Total 6289196 23.9G
Starting up this again, the kernel memory allocation grows by the size of the VM.
Code: Select all
root@infra1:~# echo "::memstat" | mdb -k
Page Summary Pages Bytes %Tot
---------------------------- ---------------- ---------------- ----
Kernel 5502694 20.9G 87% <--------------- grew again
Defdump prealloc 171182 668.6M 3%
ZFS 85196 332.7M 1%
mdb: found fewer ZFS pages than expected (2m)
Anon 170958 667.8M 3%
Exec and libs 5773 22.5M 0%
Page cache 96573 377.2M 2%
Free (cachelist) 4100 16.0M 0%
Free (freelist) 253232 989.1M 4%
Total 6289196 23.9G
Obvisously, there is some correllation between kernel memory allocation and running VMs
Re: Memory configuration
Posted: 30. Oct 2017, 20:26
by mpack
I've never used Solaris, but I'd be surprised if disk cache is allocated out of kernel RAM. In fact I'd be surprised if VirtualBox allocates disk cache at all. Normally the OS does that. VirtualBox does have a lazy / asynchronous write feature which will use RAM, but AFAIK that is allocated by a user level task (VBoxSVC) so should not be allocated out of kernel RAM.
I do wonder if I'm misunderstanding "kernel RAM" in the Solaris context though, seeing as you have 17GB allocated to it. AIUI in Windows this simply means the first 4GB of RAM, and the only thing special about it is that it's addressable by both 32bit and 64bit code, which makes it useful as a shared pool for all system processes. Btw I am not an expert on Windows kernel processes either, as I've never needed to be (not much call for it in embedded MCU work).
Re: Memory configuration
Posted: 1. Nov 2017, 08:01
by bauer40
Hi mpack,
no, I don't refer to memory allocation for Disks, I talk about memory allocation for a guest's RAM. This seems to be kernel memory, and I start asking myself wether or not Solaris maps kernel Memory to a processes virtual address space ... no, that can't be. Can it ...?
Re: Memory configuration
Posted: 1. Nov 2017, 11:08
by mpack
I think I'll have to leave you to a dev if he happens to pass by. I know too little about Solaris and what "kernel memory" means there.
Re: Memory configuration
Posted: 2. Nov 2017, 13:15
by Ramshankar
bauer40 wrote:1) do Virtual Machines started with VBoxHeadless allocate the per-guest configured memory as kernel memory, or is it "regular" memory? (of course I know a few MB kernel memory will be used by the network drivers, I care more about those 4GB virtual RAM for a VM)
2) is this memory locked in RAM, so it can't be paged out?
3) should I limit the arc cache using the mentioned procedure to make sure my VMs are held in memory?
1) The memory is allocated in the kernel but there is also parts of the VM process that use userspace allocated memory but that should be a tiny fraction in comparison to the RAM allocation of a typical VM.
2) This memory is locked down so it cannot be swapped or paged out.
3) The ZFS arc cache needs to be adjusted as recommended in the manual if required (
https://www.virtualbox.org/manual/ch12.html#idm11558)
Re: Memory configuration
Posted: 2. Nov 2017, 19:19
by bauer40
Well, thank you, but ... there is not only a little part of "normal" memory in use.
I have a guest which uses 2GB of virtual RAM - named exchange. Here is the virtual process size:
Code: Select all
root@infra1:~# grep -i memory /vbox_ufs/exchange64/exchange64.vbox
<Memory RAMSize="2048"/>
root@infra1:~# ps -efo "vsz rss pid args"|grep exchange
2257756 2155940 21323 /opt/VirtualBox/amd64/VBoxHeadless -s exchange64
You can see, the virtual process size is 2 GB, and, correct me if I'm wrong, kernel kemory is not mapped to a process.
Looking into the processes memory allocation, it looks like this:
Code: Select all
root@infra1:~# pmap -x 21323
21323: /opt/VirtualBox/amd64/VBoxHeadless -s exchange64
Address Kbytes RSS Anon Locked Mode Mapped File
0000000000400000 36 36 - - r-x---- VBoxHeadless
0000000000418000 4 4 - - rw----- VBoxHeadless
0000000000419000 53000 3448 2284 - rw----- [ heap ]
FFFF80FEB5E00000 2048 2048 2048 - rw----- [ anon ]
FFFF80FEB6200000 2048 2048 2048 - rw----- [ anon ]
FFFF80FEB6600000 2048 2048 - -rw-s---
FFFF80FEB6A00000 2048 2048 - -rw-s---
FFFF80FEB6E00000 2048 2048 - -rw-s---
FFFF80FEB7200000 2048 2048 - -rw-s---
FFFF80FEB7600000 2048 2048 - -rw-s---
FFFF80FEB7A00000 2048 2048 - -rw-s---
FFFF80FEB7E00000 2048 2048 - -rw-s---
FFFF80FEB8200000 2048 2048 - -rw-s---
FFFF80FEB8600000 2048 2048 - -rw-s---
FFFF80FEB8A00000 2048 2048 - -rw-s---
FFFF80FEB8E00000 2048 2048 - -rw-s---
FFFF80FEB9200000 2048 2048 - -rw-s---
FFFF80FEB9600000 2048 2048 - -rw-s---
FFFF80FEB9A00000 2048 2048 - -rw-s---
FFFF80FEB9E00000 2048 2048 - -rw-s---
FFFF80FEBA200000 2048 2048 - -rw-s---
FFFF80FEBA600000 2048 2048 - -rw-s---
FFFF80FEBAA00000 2048 2048 - -rw-s---
FFFF80FEBAE00000 2048 2048 - -rw-s---
FFFF80FEBB200000 2048 2048 - -rw-s---
FFFF80FEBB600000 2048 2048 - -rw-s---
FFFF80FEBBA00000 2048 2048 - -rw-s---
FFFF80FEBBE00000 2048 2048 - -rw-s---
FFFF80FEBC200000 2048 2048 - -rw-s---
FFFF80FEBC600000 2048 2048 - -rw-s---
FFFF80FEBCA00000 2048 2048 - -rw-s---
FFFF80FEBCE00000 2048 2048 - -rw-s---
FFFF80FEBD200000 2048 2048 - -rw-s---
FFFF80FEBD600000 2048 2048 - -rw-s---
FFFF80FEBDA00000 2048 2048 - -rw-s---
FFFF80FEBDE00000 2048 2048 - -rw-s---
FFFF80FEBE200000 2048 2048 - -rw-s---
FFFF80FEBE600000 2048 2048 - -rw-s---
FFFF80FEBEA00000 2048 2048 - -rw-s---
FFFF80FEBEE00000 2048 2048 - -rw-s---
FFFF80FEBF200000 2048 2048 - -rw-s---
FFFF80FEBF600000 2048 2048 - -rw-s---
FFFF80FEBFA00000 2048 2048 - -rw-s---
FFFF80FEBFE00000 2048 2048 - -rw-s---
FFFF80FEC0200000 2048 2048 - -rw-s---
FFFF80FEC0600000 2048 2048 - -rw-s---
FFFF80FEC0A00000 2048 2048 - -rw-s---
FFFF80FEC0E00000 2048 2048 - -rw-s---
FFFF80FEC1200000 2048 2048 - -rw-s---
FFFF80FEC1600000 2048 2048 - -rw-s---
FFFF80FEC1A00000 2048 2048 - -rw-s---
FFFF80FEC1E00000 2048 2048 - -rw-s---
FFFF80FEC2200000 2048 2048 - -rw-s---
There are roughly 1024 2k blocks, which sum up to 2GB.
Is this a double memory allocation, or are these non-mapped/unmarked memory blocks kernel memory?
Re: Memory configuration
Posted: 7. Nov 2017, 10:25
by michaln
bauer40 wrote:You can see, the virtual process size is 2 GB, and, correct me if I'm wrong, kernel kemory is not mapped to a process.
I'll correct you -- you're wrong
The VM's RAM is allocated as "kernel" memory on the host, but it is also mapped into the VM's process address space.