[WorksForMe] Most guests suddenly extremely slow

[popanz]

Hello,

I'm using VirtualBox since 5 years with specs:
Windows 7 (64bit) host
Disk: SSD
RAM: 16 GB
CPU: Intel Core i7-3740QM (4core/2,7GHz).
Current VirtualBox Version: 5.2.22

I'm using several guests for running old Win-XP software and for testing (max 3 VM's at the same time).
Each guest has 2 GB of RAM, which is enough for any of them.

Since yesterday, most of these guests (but not all) suddenly are extremely slow - impossible to do anything with them anymore.
...as if you would change the CPU speed for these guests down to 1% or similar.
Two of the guests are still running on normal speed.
I didn't install or change anything on the host or guest systems recently. Just shut down the one in use Win-XP VM over night, and on the next day suddenly having these problems.

When doing anything on the slow guests, their CPU usage is 100%, while it takes ages.
Usually, when you have 100% CPU in a guest, the host shows maybe 10% - 20% CPU used by VirtualBox.
Now, when having 100% CPU in the guest, the host shows 0.2% - 1.1% CPU used by VirtualBox.

Extremely slow:
2 Win-XP (32bit) guests - using them constantly every day.
1 Win7 (32bit) guest
1 Win10 (64bit) guest
1 Debian-Linux (64bit) guest

Still normal speed:
1 Win10 (32bit) guest
1 Win10 (64bit) guest - a 2 years old clone of the other Win10-64bit guest above, which is now super slow.



After reading several threads here about similar topics, I've tried:

* Return to 1 month old backup VM files.

* Upgraded to VirtualBox Version 6.0.

* Downgraded back to 5.2.22 again.

* Uninstalled / reinstalled VirtualBox (5.2.22) several times, without and with removing the settings folder ".VirtualBox".
When removing the settings folder (-> empty workspace), I've newly set up one of the Win-XP guests by just using it's .vdi-file and with the same settings as used before.

* ...also tried reinstall (5.2.22) by running setup-exe as administrator.

* Tried some different settings on the Win-XP guest, although the settings before worked nicely for years:
- 2-4 CPU cores (orig 1 CPU)
- once tried reduce CPU speed to 95% (orig 100%)
- 3D acceleration on/off (orig off)
- 2D acceleration on/off (orig on)
- IO-APIC on/off (orig off)
- PAE/NX on/off (orig off)

* On the host system, uninstalled all protection software: Spybot-Search&Destroy antimalware, Avira antivirus and Comodo firewall.
...So no protection software left on the system for most attempts.
Meanwhile installed Bitdefender instead.

* Reinstalled the VB Guest Additions on one of the Win-XP guests (took few hours at this extremely low speed).

* Removed an installed application's dll which the VBox.log marked with a "supR3HardenedErrorV" (although this supR3HardenedErrorV was there since years, with no problem ever).


On every of these listed attempts I've tried the one Win-XP guest again, but nothing helped.


Here the VBox.log and VBoxHardening.log of one complete run of the one Win-XP guest, zipped in an archive...

logs.zip

VBox.log and VBoxHardening.log

(60.09 KiB) Downloaded 10 times

I don't know whats about the multiple error messages in VBoxHardening.log about certsentry.dll (and the one message about wintab32.dll), but all these messages have been there since years, without any problem.


Meanwhile I'm really desperate and no ideas left, what I could do....
Especcially getting the two Win-XP guests to run again, is extremely important to me!
My whole work is blocked without them.
Trying to get it run since 48 hours now....

Any suggestions....?

Thanks,
Popanz

[mpack]

None of those settings changes you tried will make any positive difference whatever: once XP is installed it won't automatically adapt to any more motherboard changes. In fact most of those changes are entirely irrelevant to XP (e.g. PAE/NX).

You need to find out where the CPU time is going, then your efforts can be more directed. If multiple guests have slowed then I'd be looking at recent changes on the host, not the VMs.

[popanz]

Yeah, I'm also sure, the host is the problem.
But I didn't make any changes on the host.
There were Windows updates on Wednesday: Win7 security-rollup and .NET-Framework security-rollup. (Including Windows restart.)
But on Thursday I've been working all day with both Win-XP guests with no problem.
(Every night I'm doing a complete shut down on all guests.)
On Friday morning when trying to start the VM's the problems suddenly occured out of nothing.

There is no CPU time vanishing anywhere.
It's just like you limit the guests CPU time to a minimum in the CPU settings of the VM.
The host shows less than 1% CPU for VirtualBox while the guest is using 100% CPU inside the VM (whenever I do something in the guest).

[socratis]

A couple of things that I noticed. From the VBoxHardening.log:

5a0.21e0: \SystemRoot\System32\drivers\cfrmd.sys:
5a0.21e0:     ProductName:     Windows (R) Win 7 DDK driver
5a0.21e0:     FileDescription: Safe Deletion Driver

Can't say that I've seen that driver before. Any ideas on what its purpose is and how it may affect your system?

5a0.21e0: \SystemRoot\System32\drivers\hmd.sys:

With no more information. A quick search shows that this is part of Comodo. Does it ring a bell?

From the VBox.log, your WinXP VM and mine are almost identical. However, while you have an "Intel i7-3740QM @ 2.70GHz", and I have a "Intel Core2 Quad Q9300 @ 2.50GHz", you are getting plenty of the following messages:

00:06:28.404453 VMMDev: vmmDevHeartbeatFlatlinedTimer: Guest seems to be unresponsive. Last heartbeat received 4 seconds ago

while mine is happy. I've seen these when the host is holding back the guest.

On Friday morning when trying to start the VM's the problems suddenly occurred out of nothing.

Although I don't see any 3rd party antivirus active (except that Comodo reference), do you have anything on your host that you think could be the culprit?

[popanz]

5a0.21e0: \SystemRoot\System32\drivers\cfrmd.sys:
5a0.21e0:     ProductName:     Windows (R) Win 7 DDK driver
5a0.21e0:     FileDescription: Safe Deletion Driver

Can't say that I've seen that driver before. Any ideas on what its purpose is and how it may affect your system?

A quick google search said:
cfrmd.sys is part of COMODO System Cleaner, cfrmd.sys's description is "COMODO Safe Delete Filter"

I had the Comodo free firewall installed before.
During the fixing attempts for the VM problems, I uninstalled the firewall. (The logs are with uninstalled Comodo.)
The uninstall didn't work at first, but Comodo provides a heavy duty uninstall tool on their website, which made a successfull uninstall then.
...but obviously not totally complete....

5a0.21e0: \SystemRoot\System32\drivers\hmd.sys:

With no more information. A quick search shows that this is part of Comodo. Does it ring a bell?

Same thing - the Comodo firewall has been uninstalled before, but obviously not completely.
Would it be a good idea to simply delete these two driver files?
(If possible at all.)

From the VBox.log, your WinXP VM and mine are almost identical. However, while you have an "Intel i7-3740QM @ 2.70GHz", and I have a "Intel Core2 Quad Q9300 @ 2.50GHz", you are getting plenty of the following messages:

00:06:28.404453 VMMDev: vmmDevHeartbeatFlatlinedTimer: Guest seems to be unresponsive. Last heartbeat received 4 seconds ago

while mine is happy. I've seen these when the host is holding back the guest.

Yeah, I know. That's obviously the result from the extremely slow running VM.

On Friday morning when trying to start the VM's the problems suddenly occurred out of nothing.

Although I don't see any 3rd party antivirus active (except that Comodo reference), do you have anything on your host that you think could be the culprit?

Until Saturday I had these security applications running:
* Spybot-Search&Destroy (antimalware)
* Avira free antivirus
* Comodo free firewall.

On Saturday morning I uninstalled all of them and did all my following fixing attempts without any security software running.
The uploaded logs are also from an attempt with no security software installed.
On Saturday evening I installed Bitdefender free antivirus, since a system without security software is quite scary.
I have no other idea, which other software could cause such a problem.....

[popanz]

Things are getting more and more strange.....

Meanwhile I reactivated my 10 years old garbage computer with a normal HDD, 4 GB RAM and Windows XP Host running, using VirtualBox 4.3.40 (newer versions don't work on WinXP).
There I tried the two WinXP guests (with 256 and 1024 MB RAM).
There they are working fine, just very slow - ok, still not comparable to the current slowliness problem.
So at least having a running emergency system...

Then I made a system roll back to 5th of January (before the last Win update) on the Win7 system.
Although I was sure this should solve the problem, this roll back did not help at all....
Each VM still takes ages (maybe 30mins?) to start up and then every tiny thing happens in super slow motion - almost like freezing.

But then I discovered a super strange behaviour:
When I click on the window close button of the VM window, I get this VB dialog window asking if I wanna save the VM state, or shut down the VM through ACPI or just switch it off.
If I then choose cancel to just close that dialog window again without any action, the VM suddenly runs in normal speed for a little while!
Then on some distinct user actions inside the guest, like clicking on a link in a browser or opening a dialog in a program, it suddenly switches back to super slow.
After repeating this VM-close-dialog-action, it again runs in normal speed for a little while....

This whole thing is so strange - anyone an idea whats the problem here?

[mpack]

It would be nice to see a complete log from a session in which you did all that.

[popanz]

I somehow solved the problem!

To get consistently exchangeable VM's with my old WinXP emergency host, I also downgraded VirtualBox in the current Win7 host to version 4.3.40.
And now all guests work fine again!

Don't know what they changed after version 4, but versions 5 and 6 are not running on my Win7 anymore (although they did before for about 3 years).
So I'll stick to 4.3.40 and no updates anymore...

Thanks for your helping attempts!

[socratis]

If your idea of solving the problem is to go back in time/security, who am I to stop you...?

Marking as [WorksForMe].

[popanz]

Hihi, nice answer!
Hm, unfortunately I don't see any alternative.
Wasted 6 days on researching and trying stuff, gave up....