When running a VirtualBox VM in VirtualBox 6.1.12 on a macOS Catalina 10.15.6 host, the amount of wired kernel memory grows faster than under macOS Catalina 10.15.5, e.g. 1 GB per hour, eventually leading to a memory shortage.
Q1: macOS offers several ways to obtain memory usage information from different angles, and the results are often seemingly not consistent. Which method did you use to observe the memory usage growth rate?
I see consistent reporting from the GUI (activity monitor) and zprint for the leakage of wired pages (it's also consistent for the non-wired memory but that's not so much of an issue here).
Q2: When the VirtualBox VM is shut down, is part of the memory freed again, and if so, is the remaining memory usage growth rate significantly less?
The part of the memory that's correctly associated with the vbox instance is returned, (most of) the growth in wired memory is not returned and is a cumulative loss between sessions. Note that "sync; purge" (as root) returns the memory that's used for caching files - and can be done on a live instance with no ill-effect [other than a loss of caching performance, I suppose]. However neither quitting the VB instance nor purging the memory caches frees the leaked wired memory. The only way to recover that is a reboot
Q4: @iains: Can you try a Windows and/or Linux guest to see if it makes a difference in comparison to your macOS guests?
In due course, (I'm the main maintainer for GCC on Darwin/macOS so my machines are tied up with testing there etc. Usually I use the cfarm machines for Linux, so not much in the way of VMs around.
I don't know what's the connection between VirtualBox and the growing wired kernel memory usage you're experiencing. What I do know is that the allocation of physical memory pages for the VirtualBox VM depends on the way the guest OS touches its memory. I don't know if that correlates to your problems at all, therefore I just hinted at that.
Well, [for OSX guests] the way in which memory grows is two-fold
(1) associated with 'ramp up' of the guest ... I am doing toolchain builds which are a good stress of most machines - there's a lot of IO, memory and CPU use. My guests are mostly 4-cores, with a CPU edition compatible with the OSX era in question. Usually they have 4G RAM <= 10.7 or 6G RAM >= 10.
. The ramp-up happens quite quickly with load on the system - and saturates the VB wired memory at a little above the allocation.
(2) long-term losses .. these are mot necessarily at a constant rate, and the 1G/hour is a higher figure (when I had 3 VB instance running on the 16core machine). The rate is lower on a smaller machine with one VB instance.
I'd speculate IO buffers, screen-scraper stuff, etc. but the "losses" are accounted in the anonymous 'zones' section in the zprint output so it's hard to be sure.
Q5: Does the growth rate of the wired kernel memory correlate to the amount of physical memory pages (e.g. Real Memory in Activity Monitor, rss in ps) in use by VirtualBox somehow? Or to any other configurable memory property of VirtualBox VMs?
No that I can see directly - although as noted above I'd say it's related to the number of clients active.
iains wrote:
General question is "where do we go from here?"
.. should someone file a bug (and where ?) ?
In the VirtualBox forums, you have a broader audience to contribute observations, which is suited for discussing problems and refining the picture, but usually there are no VirtualBox developers active here. You can create a ticket in the VirtualBox Bugtracker. Provide enough information to easily reproduce the problem, and add a link to this thread. Then hope for the bug ticket to catch the interest of a VirtualBox developer ...
The post mentioning VMware reinforces the probability that this is a virtualisation framework bug - but I'd say that a bug reported from the VB developers is going to be more successful than my anonymous panic reports in getting a fix ....
@fth0 : I'm doing a new run at the moment keeping the output from zprint at various points - when that's done, I'll attach the wired portion at least, (a full test cycle takes most of a day so do't hold your breath
)