VBoxHeadless reliably eating 100% CPU

[michaln]

Sampling the process seems to hang, as shown in the attachment.

Edit: Attached the Activity Monitor stats for the process, in case it's at all helpful.

It's not. I guess we're stuck here because you are unable to get the data we need. Best thing you can do is figure out what it is that's so unusual you're doing (and you are doing something unusual or there would be lots more identical complaints).

[Perryg]

Sampling the process seems to hang, as shown in the attachment.

Edit: Attached the Activity Monitor stats for the process, in case it's at all helpful.

It's not. I guess we're stuck here because you are unable to get the data we need. Best thing you can do is figure out what it is that's so unusual you're doing (and you are doing something unusual or there would be lots more identical complaints).

Vagrant?

[harveyr]

EDIT: Ignore this one. It looks like these spikes are caused by visible activity in the guest.

I was able to get a sample by running it from the command line with sudo. It's attached.

This time, VBoxHeadless seemed to be reliably bouncing between ~80% and ~150% CPU in Activity Monitor.

[harveyr]

Okay, here's another sample, this time under the same symptoms as originally described: ~100% CPU in Activity Monitor, and 0-1% in the guest.

[michaln]

I can't tell from here if it's normal behavior or a bug. Just to be clear, does the CPU spike go away after some time? Or does it persist even when whatever the guest OS is doing is done?

What I can tell is that there is a lot of networking activity in the VM and the NAT thread is quite busy. So that's something. You have much better idea than anyone else what the guest OS is actually doing.

Please remind me, how exactly are you determining the CPU usage in the guest? And does 100% load constitute 1 CPU or all four? Depending on what exactly is going on, it is possible that there are lots of network packets flying around and while the guest CPU isn't too busy sending/receiving them, the hypervisor has to push them in an out of the VM. So the guest seems mostly idle but it's triggering lots of activity.

[harveyr]

Just to be clear, does the CPU spike go away after some time? Or does it persist even when whatever the guest OS is doing is done?

It persists, hopping between about 98% and 103%, which I believe is just one core on the host. I haven't seen it drop below that. I haven't waited hours, but I've waited tens of minutes.

What I can tell is that there is a lot of networking activity in the VM and the NAT thread is quite busy. So that's something. You have much better idea than anyone else what the guest OS is actually doing.

Here's a bit more on what we're doing and what triggers this condition. We're running a few services in the guest: (1) a Python/Django website, (2) a collection of simpler Python services that respond to HTTP and protobuf requests, and (3) a Node service that bundles JavaScript and serves it over HTTP using browserify and budo. Number 1 uses Postgres and Memcached. The others run in memory.

Using Vagrant, we're mounting a few directories on the guest via NFS and forwarding a few ports to the guest.

The condition starts as soon as I start up the guest and view that first website in the browser, which causes requests to be sent to all three services. However, the condition persists even after I shut everything down in the guest—those three services, memcached, postgres, supervisord, and everything else I can see in top.

Please remind me, how exactly are you determining the CPU usage in the guest? And does 100% load constitute 1 CPU or all four? Depending on what exactly is going on, it is possible that there are lots of network packets flying around and while the guest CPU isn't too busy sending/receiving them, the hypervisor has to push them in an out of the VM. So the guest seems mostly idle but it's triggering lots of activity.

I'm using top/htop to view guest activity, and mentioned just above, it's just one core on the host that's been consumed.

I really appreciate your time on this! I'm going to experiment with disabling things on startup to see if one of the services described above is triggering network grind.

[michaln]

Is the condition relatively easy to trigger? In other words, would it be possible for you to provide a/the VM (exported as OVA appliance, no vagrant or anything) with some simple way to trigger the misbehavior?

To prove or disprove the theory that NAT is the problem you could try switching the VM to bridged or host-only networking or possibly NAT Network. That will obviously change how the VM interacts with the network, but it will get the current prime suspect out of the picture.

[harveyr]

Unfortunately, there's enough sensitive code on there that providing a sanitized export for you folks would be exceedingly difficult. (Edit: Though, if I'm running out of things to try, perhaps I can try to trigger the behavior with some dummy code and get that to you.)

However, I will definitely try the alternative networking configs you mentioned as soon as I have some spare minutes. Thank you!

[harveyr]

A quick update on this: We're still investigating this issue in between higher priorities. We have a hunch that one of the services we're running in the guest is triggering the problem. But the evidence is still anecdotal.

We'll get back here with an update when we have something to go on. Thanks!

[michaln]

Thanks for keeping us posted.

[cphc]

I just want to chime in and say i have the exact same problem, on the exact same computer. (Macbook Pro i7 : Vagrant Ubuntu - from Laravel Homestead).

It seems also from googling around, that this has been a problem for a bunch of other people. I have experienced the problem sporadically for a while.

For me the cpu usage "randomly" gets stuck at 50%.
I also suspect it has todo with some process running on the server. I run Node that runs some Grunt watchers for example.

The weird part is that it happens very much at random. Like i am not actively changing anything affecting the service running. My computer can be idle, or i can be editing some text or something completely unrelated and suddenly, 50% cpu usage and my fans go on.

My intuition tells me it could be some kind of feedback loop created by some maintenance/backup/logging process that messes with the the way Vagrant syncs folders from the Host system to the VM.

[mightyiam]

+1 on this.

Also Vagrant, Django, memcached and a bunch of other stuff.

[mightyiam]

I had a crash, perhaps as a result of this.

[mightyiam]

Attached is a screenshot showing that the offending thread is "NAT".

[mightyiam]

Oh, and I'm on 5.0.0.