Emacs garbage collection too slow. Maybe due to the unexpectely small amount of VB host memory usage

Discussions about using Linux guests in VirtualBox.
Post Reply
zephyrus
Posts: 9
Joined: 16. Oct 2009, 10:47
Primary OS: MS Windows XP
VBox Version: PUEL
Guest OSses: linux solaris

Emacs garbage collection too slow. Maybe due to the unexpectely small amount of VB host memory usage

Post by zephyrus »

Hi,
I am using VBox under Windows 10 Pro host, and run Debian GNU/Linux guest inside it.

I wonder if someone sees a similar problem I experience and figures out how to solve it.

My problem is this. I see sometimes the garbage collection within Emacs running under linux guest is extraordinarily slow. Emacs uses so-called Emacs-Lisp interpreter within it, and being a lisp-variant, it needs to invoke garbage collection from time to time.
I never got bothered by garbage collection time until it became rather long maybe 5 years ago or so.

It could be that there could have been a change of GC-related routine in Emacs, but the slowness I see cannot be explained with such a change.
I tried to see if there had been any such change in the Emacs code, but I could not find one. (Not that my search was complete, but there was no immediately found smoking gun.)

Then, it took me quite a while but it dawned on me that there is a problem on the host side.
Now I am a bit perplexed that VBox seems to use much smaller host memory (under windows 10 pro) than the amount of memory I assign to the guest linux.

I have 32GB of physical memory on the PC, and allocate 16GB to linxu guest inside VBox.

Naturally, I expected VBox to allocate that amount of host physical memory or close to 16GB anyway to the linux guest.

But looking at the memory usage using Windows 10' task manager, the memory usage of virtualbox virtualmachine is rather small. Too small for my purpose.

Right now, in the descending order of memory usage, the list is as follows. (The amount changes dynamically, and I am typing it by hand, so it is only a crude approximation.)
1. 4,201 MB Forefpx
2. 584 MB OpenJDK platform binary (I run OmegaT that uses JDK)
3. 370 MB Google Chrome
4. 334 MB Slack
5. 234 MB Everything
6. 196 MB Skype
7. 146 MB Thunderbird
8. 127 MB Desktop window mangaer (of Windows 10, I suppose)
9. 111 MB VirtualBox Virtual Machine
10. 102 MB ownCloud

You see the largest user of memory is Firefox at 4+ GB. Virtual Box Virtual Machine uses only 111MB (!?).
Isn't the allocated memory by VBox Virtual Machine (No. 9 in the above list) tad too small, considering 16GB allocated memory to the linux guest?

I *THINK* the emacs garbage collection sometimes gets very slow due to its random memory reference pattern while VBox may be trying to do its own virtual machine memory allocation/purge from the host physical memory.
Unless such memory access handling is done, there is no way VBox can support a guest with 16GB memory assigned to it.
Usually, the locality of reference often works to the advantage of VBox and I have not really suffered from the slowness of I/O and memory usage. Even the CPU and I/O intensive operation of building thunderbird mail client works like charm within the linux guest.
But this garbage collection that obviously violates the locality of reference suffers badly.

But maybe I am misguided. The 111MB reported by task manager under windows 10 may not be quite right.
It looks way too small.
The reason I say this is that task manager says 95% of memory is used up, but I don't believe adding the numbers in the memory column do not add up to close to 32GB at all.

Inside the linux guest, emacs uses about 1.2 GB resident memory and 2.4GB virtual memory according to "top" command output.
Naturally I think if VBox at least uses more host memory than 2.4GB, the chances of emacs memory pages are already within the host (virtual?) memory is large and garbage collection should be faster (but of course, the speed depends on how much memory space of running emacs image is resident eventually in the VBox virtual machine's host memory.)
My point is that it seems as if VBox VM is not mapping the guest VM memory space to the host memory space as efficiently as possible.
This of course may be intentionally done to run reasonably well-behaved VM guests on a machine at the same time.

I would say, emacs with large memory space when it invokes garbage collection is not a well-behaved client in terms of memory reference pattern.

My question is I wonder if there is a way to forcibly let VBox to use up more windows 10 host memory (by giving a certain number, say, 4GB or something). I believe that will streamline the sometimes very sluggish Emacs garbage collection behavior. (Sometimes, GC continues close to a few minutes!)

If anyone knows how to do it or any insight into the problem I see, I appreciate it very much.

TIA
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Emacs garbage collection too slow. Maybe due to the unexpectely small amount of VB host memory u

Post by fth0 »

I cannot tell you why the Emacs GC is slow in your case (although I might have some ideas, see below), but I can explain a few things you've been wondering about:

1. The Windows Task Manager can only give a superficial view of the Windows Memory Management. If you really wanted to learn about that, you'd need to use for example Sysinternals VMMap (Virtual Memory) and RAMMap (Physical Memory).

2. Type-2 hypervisors like VirtualBox use so-called Driver Locked memory, which is not attributed to the VirtualBoxVM process. That's why you cannot see this memory in the Windows Task Manager, but only in RAMMap.

3. While a Windows guest OS accesses all its memory during startup, a Linux guest OS accesses it only when it needs it. In consequence, your Linux VM won't use the whole memory in the beginning, but a gradually increasing amount of it. Note that the memory is never given back during the runtime of the VM.

4. If you want to force the VM to allocate its whole memory in the beginning for a test, you can use VBoxManage setextradata "VM name" "VBoxInternal/RamPreAlloc" "1". Repeat the command without the "1" at the end to undo it. Execute the commands while the VM is not running.

Regarding your initial problem, it's conceivable that your Windows host OS is running under the Hyper-V hypervisor. In the status line of the Linux VM's window, do you see a green turtle or a blue chip icon with a "V" on its back?
zephyrus
Posts: 9
Joined: 16. Oct 2009, 10:47
Primary OS: MS Windows XP
VBox Version: PUEL
Guest OSses: linux solaris

Re: Emacs garbage collection too slow. Maybe due to the unexpectely small amount of VB host memory u

Post by zephyrus »

Thank you for the response.
fth0 wrote:I cannot tell you why the Emacs GC is slow in your case (although I might have some ideas, see below), but I can explain a few things you've been wondering about:

1. The Windows Task Manager can only give a superficial view of the Windows Memory Management. If you really wanted to learn about that, you'd need to use for example Sysinternals VMMap (Virtual Memory) and RAMMap (Physical Memory).
This is a news. Thank you for the tips. I downloaded sysinternals.

fth0 wrote: 2. Type-2 hypervisors like VirtualBox use so-called Driver Locked memory, which is not attributed to the VirtualBoxVM process. That's why you cannot see this memory in the Windows Task Manager, but only in RAMMap.
rammap shows that there is indeed driver locked memory, which is 1^+GiB. So this must be the memory used by virtualbox VM.

fth0 wrote: 3. While a Windows guest OS accesses all its memory during startup, a Linux guest OS accesses it only when it needs it. In consequence, your Linux VM won't use the whole memory in the beginning, but a gradually increasing amount of it. Note that the memory is never given back during the runtime of the VM.

4. If you want to force the VM to allocate its whole memory in the beginning for a test, you can use VBoxManage setextradata "VM name" "VBoxInternal/RamPreAlloc" "1". Repeat the command without the "1" at the end to undo it. Execute the commands while the VM is not running.
I will try this later to see if this improves my situation.
fth0 wrote: Regarding your initial problem, it's conceivable that your Windows host OS is running under the Hyper-V hypervisor. In the status line of the Linux VM's window, do you see a green turtle or a blue chip icon with a "V" on its back?
No. I was bitten by Hyper-V last year. I happily installed sandbox feature of Windows 10 pro thinking it would be handy to check the access to dubious sites that may want to force the downloading of malware, etc.
Downloading it automatically turned on Hyper-V. After that, I noticed very long I/O wait in the gust linux (I am monitoring the longish build of thunderbird mail client using xosview.)
After google search regarding the strange slowdown, I figured it was Hyper-V and so turned it off.

Now, I am looking at rammap listing.
One thing that makes me wonder is the large page table size for VirtualBoxVM.exe (see attached).
Considering that we do have 16GB VM (in the driver locked area) assigned to VBox, it is understandable that the page table is large.
But I wonder how this page table itself is managed. I need to understand the way VBox manages the
host physical memory (and virtual memory) to emulate the guest's emulated real memory.
I think the algorithm used currently probably works OK when the locality of reference is good, but if an ill-behaving program (in terms of memory access pattern) tries to access large area of memory in the guest, the algorithm may not be that efficient.
That is a pure guess.
I have to somehow figure this out to avoid the costly longish GC time. (It is a hurdle to user productivity. I want to clear the shell buffer to start afresh, and then this 2-3 minutes GC kicks in and I can't do a thing.)
I need to check if there has been any change in the low-level memory allocator of emacs, especially string GC, which seems to take up most of the time.

Again, thank you for your tips.
It helps me to obtain some numbers which had not been visible to me before.
Attachments
rammap processes view. Large page table for VirtualBoxVM.exe
rammap processes view. Large page table for VirtualBoxVM.exe
large-page-map.PNG (90.12 KiB) Viewed 1626 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Emacs garbage collection too slow. Maybe due to the unexpectely small amount of VB host memory u

Post by fth0 »

zephyrus wrote:Thank you for the response. [...] This is a news.
BTW, I've just found an older thread of yours, and you seem to have missed the answer(s): Is allocated memory to VBox "locked" in real memory? It should, but it does not seem to be. ;)

FWIW, I've also read the thread in the GNU Emacs bug list that you've linked to.
zephyrus wrote:I need to understand the way VBox manages the host physical memory (and virtual memory) to emulate the guest's emulated real memory.
Depending on your previous knowledge of memory management internals, that would be really a lot to learn: Topics are the physical memory management of the host OS, (not) the virtual memory management of the host OS, the physical memory management of the guest OS, the virtual memory management of the guest OS, nested paging support of Intels/AMDs hardware virtualization (EPT/RVI/SLAT), and finally the VirtualBox implementation of connecting all that.

But perhaps this is not necessary. I'm not convinced that you're on the right track (although it's possible), because I see at least two more directions to go:

Regarding locality of reference, CPU cores and caches (I know you mentioned excessive cache misses before) could play a role. From the point of view of the host OS, the VM's vCPUs are nothing more but ordinary host OS threads of the VirtualBoxVM process, and the host OS will schedule them at its own discretion. In consequence, they can "jump around" multiple times per second, possibly thrashing the L1/L2 caches of the host CPU.

Regarding VirtualBox, there's a lot of information in the VBox.log files, especially for people being able to interpret them. ;) I'm interested in seeing a VBox.log file from a VM run where the issue occurred (the last 4 rotated log files are kept). Please shut down the VM (or at least close it) before providing the file, because there are a lot of statistics inside then. If the issue didn't occur during the last 4 VM runs, please provide a VBox.log file anyway as a baseline. Often enough the real problem is visible nonetheless.
Post Reply