[Fixed by supplemental update] kernel panic from memory leak on 10.15.6

[iains]

two different systems
* Mac Pro 7,1 (2019) 16 core Xeon-W with 96G RAM
* Mac mini 6 core i5 with 64G RAM
catalina 10.15.6
VB 6.1.12

guest(s) are earlier versions of OSX (I use the Box installations for testing GCC toolchain builds across the OSX range).

Monitoring I see around 1Gb / hour kernel memory leakage (wired) which eventually results in spontaneous quits of GUI programs and a kernel panic. The panic is from some random process (whatever happens to make the request that exceeds the resource) - so there's nothing helpful in terms of a VB backtrace (nor is there anything useful in the visible allocations for the VB guests - those don't change after the initial ramp up).

Of course, this could be a virtualisation framework bug in 10.15 * the same setups work fine on 10.14 and 10.13. *

Any ideas / known issue / workaround?

[david0511]

I had exactly same issue, downgraded to Catalina 10.15.5 then the problem was resolved, fortunately I made a clone backup before catalina patch updated.

[iains]

The problem is that the leak is in wired kernel memory (so that quitting the app/client, purging memory or even logging out makes no difference) the effect is cumulative.

I've reported the kernel panic through the usual channel to Apple - but of course it's hard to tell if the bug is with OS virtualization facilities, or is in the vbox client impl.

[yyc]

confirmed on two systems, both running catalina 10.15.6 and virtualbox 6.1.12:
* 2013 imac, i7 32gb
* 2019 imac, i9 64gb

additional observations:
issue/symptoms match exactly what @iains has described
memory leak for me occurs only with windows 10 (2004 update) guest; it does not manifest with manjaro 18 guest
setup worked fine on 10.15.5
downgrading virtualbox to 6.1.10 on catalina 10.15.6 does not resolve the issue

[brazune]

Not sure if it's the same problem, but I'm getting also daily system crashes since the upgrade to Catalina 10.15.6 with Version 6.1.12.

Code: Select all

Process:               VBoxSVC [5162]
Path:                  /Applications/VirtualBox.app/Contents/MacOS/VBoxSVC
Identifier:            VBoxSVC
Version:               0
Code Type:             X86-64 (Native)
Parent Process:        ??? [1]
Responsible:           Terminal [964]
User ID:               502

Date/Time:             2020-07-24 12:41:41.360 +0100
OS Version:            Mac OS X 10.15.6 (19G73)
Report Version:        12
Bridge OS Version:     4.6 (17P6065)
Anonymous UUID:        A77B5A87-1F66-4FA6-14FB-C0E2148B5541

Sleep/Wake UUID:       C3AD3BC8-0E31-4BD9-90CB-9712C52351F5

Time Awake Since Boot: 13000 seconds
Time Since Wake:       6100 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BREAKPOINT (SIGTRAP)
Exception Codes:       0x0000000000000002, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Trace/BPT trap: 5
Termination Reason:    Namespace SIGNAL, Code 0x5
Terminating Process:   exc handler [5162]

[iains]

Not sure if it's the same problem, but I'm getting also daily system crashes since the upgrade to Catalina 10.15.6 with Version 6.1.12.

It's very hard to tell - since the effect of the memory leak is to destabilise the system, however I'd make these observations:

* that the client stability is not particularly affected by this issue - it's the host-side that is crashing (In fact, on some occasions the client VB was running on quite happily after the host had become unusable)

* If you are seeing the same crash each time, it seems less likely to be this issue (characterised for me by some random process failing when it attempts to make a memory allocation).

General question is "where do we go from here?"
.. should someone file a bug (and where ?) ?
(the crash reports from the kernel panics have gone to Apple - but that's an anonymous thing, so there will be no feedback unless/until there's a new patch release (if it is their bug)).

[Heliax]

Hi,

Not sure if it's the same problem too, but I've got at least four Kernel Panic/System Crash and auto-reboot of my Mac Mini (16Gb) running two small VM (Ubuntu Server, 2 and 4 Gb of RAM) since upgraded to Catalina 10.15.6 and VB 6.1.12.
Didn't have any issues for years, before upgrading to these versions.

Regards,

[fth0]

Any ideas?

You could try and investigate the kernel memory usage using

Code: Select all

sudo zprint -t -s

---

memory leak for me occurs only with windows 10 (2004 update) guest; it does not manifest with manjaro 18 guest

Note that a Windows OS will touch all available memory right during startup, whereas a Linux OS will only touch the memory when needed, which will also be all of the available memory after some hours. In consequence, the VirtualBox lazy memory allocation on the host only shows up on Linux guests.

[yyc]

memory leak for me occurs only with windows 10 (2004 update) guest; it does not manifest with manjaro 18 guest

Note that a Windows OS will touch all available memory right during startup, whereas a Linux OS will only touch the memory when needed, which will also be all of the available memory after some hours. In consequence, the VirtualBox lazy memory allocation on the host only shows up on Linux guests.

since my last post i have let both host machines run for some 20 hours, after a cold re-boot, starting only virtualbox with one guest on each machine. one host ran with a manjaro 18 guest, the other with a windows 10 guest. neither machine (on host or guest side) was actively used for anything else during that 20 hour period.
the result was as follows: both machines leaked (wired) kernel memory as decribed by @iains, but the host with the windows 10 guest did so at a much faster rate. after 20 hours, the host running the windows 10 guest had consumed nearly 30gb of extra (wired) memory while the host running the linux guest consumed less than 2gb of additional (wired) memory. roughly the rate of leaking memory on the host differed by about 15 to 1, i.e. windows vs linux guest.
definitely that is a correction to my observation yesterday that linux guests were not affected. i simply did not notice the much slower memory leak on the host with the manjaro guest, which was being actively used by other applications and services at the time. i am curious if multiple guests would compound the rate at which memory is leaked on the host, but that's a test for another day...
i am not sure i understand the last sentence in your post and i hope you can clarify: it is not the initial memory allocation that is the problem; rather the slow leaking of (wired) kernel memory on the host that continues until all memory is exhausted and results in a kernel panic on the host. actively using or not using the guest after it is launched does not seem to be a factor.

[iains]

Code: Select all

sudo zprint -t -s

Not terribly illuminating, sadly.

concentrating on the leak of wired memory, since that's the problem here.

What this showed is that the allocation attributed to the vbox driver is not changing - good.
There are a few modest increases in other drivers uses (but not significant).

All the lost memory is in the 'zones' pool - so there's no real way to point at the culprit from that output - someone with a kernel debug setup probably needs to run this under a leaks detection environment.

[fth0]

I'll try and summarize what I understand so far from reading your posts, and add my own thoughts and ask some questions:

When running a VirtualBox VM in VirtualBox 6.1.12 on a macOS Catalina 10.15.6 host, the amount of wired kernel memory grows faster than under macOS Catalina 10.15.5, e.g. 1 GB per hour, eventually leading to a memory shortage.

Q1: macOS offers several ways to obtain memory usage information from different angles, and the results are often seemingly not consistent. Which method did you use to observe the memory usage growth rate?

Q2: When the VirtualBox VM is shut down, is part of the memory freed again, and if so, is the remaining memory usage growth rate significantly less?

Using a Windows guest, the memory usage grows faster than when using a Linux guest.

Q3: @yyc: You used two different (albeit similar) hosts to examine that. Can you cross check and confirm that the guest OS makes the difference, and not the host?

Q4: @iains: Can you try a Windows and/or Linux guest to see if it makes a difference in comparison to your macOS guests?

I don't know what's the connection between VirtualBox and the growing wired kernel memory usage you're experiencing. What I do know is that the allocation of physical memory pages for the VirtualBox VM depends on the way the guest OS touches its memory. I don't know if that correlates to your problems at all, therefore I just hinted at that.

Q5: Does the growth rate of the wired kernel memory correlate to the amount of physical memory pages (e.g. Real Memory in Activity Monitor, rss in ps) in use by VirtualBox somehow? Or to any other configurable memory property of VirtualBox VMs?

General question is "where do we go from here?"
.. should someone file a bug (and where ?) ?

In the VirtualBox forums, you have a broader audience to contribute observations, which is suited for discussing problems and refining the picture, but usually there are no VirtualBox developers active here. You can create a ticket in the VirtualBox Bugtracker. Provide enough information to easily reproduce the problem, and add a link to this thread. Then hope for the bug ticket to catch the interest of a VirtualBox developer ...

[fth0]

All the lost memory is in the 'zones' pool

I'd be interested in taking a look myself. Can you provide two complete outputs of the command, where the difference is 1 GB (or several 100 MB at least)?

[xbuntugest]

there are 10.15.6 implosions also happening to vmware users, sorry can't post link but there's multiple threads in their community forums

without the benefit/skill of looking under the hood i found the uptime greatly improved by turning off file sharing on the host, and induced a kernel panic in minutes by updating the 'macos server' software

is it possible this is related to virtualization due to higher memory pressure or some such, and not virtualization itself

adding: is it possible, for instance, that macos memory management is misconfigured

[yyc]

without the benefit/skill of looking under the hood i found the uptime greatly improved by turning off file sharing on the host, and induced a kernel panic in minutes by updating the 'macos server' software

thanks @xbuntugest ... file sharing is disabled by default on both host machines i use.

[yyc]

thanks for looking into this issue further and providing feedback...

When running a VirtualBox VM in VirtualBox 6.1.12 on a macOS Catalina 10.15.6 host, the amount of wired kernel memory grows faster than under macOS Catalina 10.15.5, e.g. 1 GB per hour, eventually leading to a memory shortage.

correct. leaking wired kernel memory on the host was not an issue under catalina 10.15.5 with any version of virtualbox i have used, including and up to virtualbox 6.1.12. subsequently downgrading virtualbox to, say, 6.1.10 does not resolve the problem.

Q1: macOS offers several ways to obtain memory usage information from different angles, and the results are often seemingly not consistent. Which method did you use to observe the memory usage growth rate?

i used Activity Monitor, observing memory usage increasing in "Memory Used" and "Wired Memory". i understand that this is normal behaviour, except for the fact that memory consumption increases steadily and relentlessly until all available memory is consumed, resulting in a kernel panic on the host.

Q2: When the VirtualBox VM is shut down, is part of the memory freed again, and if so, is the remaining memory usage growth rate significantly less?

no, wired kernel memory is not freed again after virtualbox is shut down, but memory usage growth rate returns to normal levels.

Using a Windows guest, the memory usage grows faster than when using a Linux guest.

yes. i am seeing a difference that is approximately 15 to 1, with a windows guest excessively consuming kernel memory on the host much faster than a linux guest. this behaviour has been consistent on both host machines, with identical guests, after several re-starts etc.

Q3: @yyc: You used two different (albeit similar) hosts to examine that. Can you cross check and confirm that the guest OS makes the difference, and not the host?

checked. the host makes no difference; behaviour is exactly the same.

Q5: Does the growth rate of the wired kernel memory correlate to the amount of physical memory pages (e.g. Real Memory in Activity Monitor, rss in ps) in use by VirtualBox somehow? Or to any other configurable memory property of VirtualBox VMs?

i do not believe so. however, if you have a particular "hunch" you would like to test, let me know and i can probably run something overnight.