Powering down Linux guest taking a long time

[MatsBoekelund]

Hi. This post might fit better under "Windows hosts", depending on what the root cause of it is... I am no longer seeing this issue but would very much like to understand what it is/was caused by since a colleague of mine has reported similar behavior. (and it disappeared without my doing anything tangible to fix it, at least conciously, so it will probably come back) Originally I thought this was caused by my starting to use VBox 6.1.12 recently (was on 6.0.X before) but I doubt that is the cause. The two attachments contain logs and .vbox files for two scenarios, one "OK" and one where the problem occurs. The issue seems to be something going wrong with the disk when powering down, causing the power-down to take minutes. Here an excerpt of something I saw in the log file:
00:01:32.818374 0078 - 00000000 0000f400 - base=00000000 limit=00000000 dpl=3 DataDownRO Present 16-bit
00:01:32.818376 ************** End of Guest state at power off ***************
00:05:00.625582 PDMR3PowerOff: Driver 'VD'/0 on LUN#0 of device 'piix3ide'/0 took 207 807 119 746 ns to power off
00:05:00.654401 Audio: Disabling input for driver 'DSoundAudio'
00:05:00.654539 Audio Mixer: Setting recording source of sink '[Recording] Line In' to '<None>'

as opposed to:
00:06:12.539888 ************** End of Guest state at power off ***************
00:06:12.589498 Audio: Disabling input for driver 'DSoundAudio'
00:06:12.589730 Audio Mixer: Setting recording source of sink '[Recording] Line In' to '<None>'
00:06:12.589897 Audio Mixer: Setting recording source of sink '[Recording] Microphone In' to '<None>'
00:06:12.677789 PDMR3PowerOff: 137 861 592 ns run time
00:06:12.677863 Changing the VM state from 'POWERING_OFF' to 'OFF'

As of yesterday I am no longer seeing this problem. One thing I did yesterday was to (successfully) run a VM that I had created in VBox 6.0.4 on another PC. Somehow I find it hard to believe that this action would "fix" something that can't be resolved by numerous reboots of the host though. (Windows 10, Dell 7280 laptop). The guest is CentOS 7.2 Linux.

All that was done to collect the two sets of attached logs was booting the guest and powering it down using "powerdown -P now"

Any help would be kindly appreciated. Thx in advance + best regards / Mats
PS: The attachments were created with VBox 6.1.2 which I tried only temporarily, I am now back on 6.1.12. DS

[birdie]

Try changing the chipset type from PIIX3 to ICH9.

[MatsBoekelund]

Hi birdie, thx.
I will pass this on to colleagues who have seen the same issue. Personally I no longer see it and have no explanation a what may have provided a fix so I think it's likely to re-surface. Thx + BR / Mats

[MatsBoekelund]

Hi birdie (and anyone else reading this;-)
I just saw this happening again, also with the other chipset, it happens on every shutdown once it has started to happen:
00:02:08.404218 ************** End of Guest state at power off ***************
00:04:33.178131 PDMR3PowerOff: Driver 'VD'/0 on LUN#0 of device 'ahci'/0 took 144 773 861 066 ns to power off
00:04:33.184618 Audio: Disabling input for driver 'DSoundAudio'
00:04:33.184663 Audio Mixer: Setting recording source of sink '[Recording] Line In' to '<None>'

I power cycled this VM multiple times before it started to happen. Between it not happening and happening, I did select the "Enable PAE/NX" box in System -> Processor settings, I doubt that's what caused it though since as far as I can tell, I've had that selected on all similar VMs, it was just off here for trial purposes regarding another issue. Besides, I have since unticked the box again and it still happens.

Any clues, anyone?
thx + BR / Mats

[scottgus1]

I think that toggling settings from what the New Guest Wizard would pick for the guest OS type is likely to bung things up.

Make a new guest of the correct type, don't bother making a disk for it. You just want to see what the settings are supposed to be. These settings should be used by the guest you are having trouble with. Note that increasing processor count and Ram/Video RAM are normal. Changing other system settings may alter/degrade/kill-off performance.

FWIW ICH9 is reported as experimental in the manual, and "Enable PAE/NX" seems to be related to memory not disk control, so I don't know how that would have helped.

Please restore settings to the New Guest Wizard's defaults, with allowable processor count and Ram/Video RAM adjustments. Run until you see the problem happen again, then zip and post that run's vbox.log and the host's VboxSVC.log. Also zip and post that guest's .vbox file.

[birdie]

This looks like a bug in VirtualBox itself. You might want to file a bug report but it's not worth it unless you can reveal how it can be easily and reliably reproduced.

[scottgus1]

If this was a bug (and I'm not saying it isn't) it is manifesting itself rather locally. The forum hasn't blown up with it. Meaning it is more likely that something local is the cause.

A bug that can be successfully dealt with on Bugtracker needs a reliable reproduction scenario on a plain-vanilla setup. Tweaking settings is not a plain-vanilla setup and has to be reversed.

[fth0]

I did select the "Enable PAE/NX" box in System -> Processor settings

This CPU setting is only used for 32-bit guests, and ignored for 64-bit guests, which have the corresponding functionalities anyway.

Regarding your problem, you could take a look at the Windows Task Manager, if there are any read or write accesses to the host disk. And if the VM window is still visible during those 2+ minutes, you could also watch the hard disk icon.

[MatsBoekelund]

Hi scottgus1, birdie and fth0, thanks for your inputs:-)
Without a way of reproducing it I would not bother filing a bug report. Even so, it's not really a show-stopper, more an annoyance. The main reason I want to chase it down and understand why it happens is that it seems indicative of something not being right which may cause other problems.

Like in another post, my apologies for not having been able to work on this topic much in the last week, I am trying to get back on it. A colleague of mine on a different continent has the same issue with a Guest I set up for him, that Guest did originally not show the slow powerdown when I ran it but started to do so after a cycle of starting/powering down. (I have no reason to believe that the changed PAE/NX settings had any real effect)

Another issue I'm currently looking at is errors seen when installing the GuestAdditions. The GAs seem operational, also in terms of display resizing, for example, but a backup logfile in /var/log points to tons of errors having to do with the X-stuff. (Display) Depending on what I find on that issue, there may be an additional post coming, I don't believe the two are related. (fth0, maybe I just need to remove the headset and the fog will be gone;-)

Will try the suggestions provided here and update.
thx + BR / Mats

[MatsBoekelund]

I did select the "Enable PAE/NX" box in System -> Processor settings

This CPU setting is only used for 32-bit guests, and ignored for 64-bit guests, which have the corresponding functionalities anyway.

Regarding your problem, you could take a look at the Windows Task Manager, if there are any read or write accesses to the host disk. And if the VM window is still visible during those 2+ minutes, you could also watch the hard disk icon.

Hi fth0, thanks, I noticed this started to happen on another guest just a couple of hours ago and had a look at the Windows Task Manager during the powerdown-period: Indeed, there's fierce reading activity during the time it takes, much more intense than when I actually use the Guest for what it's intended to do. (I attached a picture) That reading activity drops immediately as the Guest window closes.

On a side note, troubleshooting a different problem I have inside my CentOS 7.2 guests, I have dropped back to VBox version 6.0.24 and the Guest I used today I created last night (also in 6.0.24) on my PC at home. The reason for dropping back to 6.0.24 from 6.1.12 was to try and rule out any issues with hw-virtualization, thinking that this may cause these Guests to be slow. A general issue as of late is that the Guests are performing tasks quite slowly, on one Guest I had installed sysbench and performed a couple of quick tests, particularly CPU and disk related but that I saw no smoking gun. (then again, I don't have any real data to compare with) Run on my PC at home, the current Guest in question seemed to perform fine but not on my laptop. My next step for that particular issue will be to try the same Guest on a different PC here in the office.

Since I am seeing the Guests having sluggish performance also in 6.0.24 I guess that is not caused by my going to 6.1.X, probably a good thing. While colleagues have also seen the delayed shutdown, I don't think they've seen the sluggishness I'm seeing, maybe suggesting that is specific to my laptop.

I started looking at Windows log files but will need to ask our IT if they can read something out of them, I can't really tell good from bad.

Talking about log files, I noticed several lines in the Hardening-log similar to this one:
201c.2dd8: *00007ffc7cd80000-00007ffc7cd80fff 0x0002/0x0080 0x1000000 \Device\HarddiskVolume3\Windows\System32\advapi32.dll
201c.2dd8: supHardNtVpScanVirtualMemory: Ignoring unknown mem at 00007ffc7cd80000 LB 0x1000 (base 00007ffc7cd80000) - 'advapi32.dll'
201c.2dd8: 00007ffc7cd81000-00007ffc7cddffff 0x0020/0x0080 0x1000000 \Device\HarddiskVolume3\Windows\System32\advapi32.dll
201c.2dd8: supHardNtVpScanVirtualMemory: Ignoring unknown mem at 00007ffc7cd81000 LB 0x5f000 (base 00007ffc7cd80000) - 'advapi32.dll'
201c.2dd8: 00007ffc7cde0000-00007ffc7ce14fff 0x0002/0x0080 0x1000000 \Device\HarddiskVolume3\Windows\System32\advapi32.dll

Not really being able to tell if that means anything or not, I attached the VBox.log and VBoxHardening.log from my latest session on my laptop.

thx + BR / Mats

[MatsBoekelund]

Just a brief update: A few minutes ago I tried a copy of the same Guest, using a Win10 Desktop PC here in the office as a host, also running VBox 6.0.24. There it also runs our software very very slowly whereas the same Guest seems to run fine on my home PC. Said Desktop is an 8-Core 3.3GHz Dell pC, so other than Dell, a common denomiator could be the Antvirus stuff? (Cortex, in our case) I will ask IT if an exception can be made for files related to VBox. (the home PC is a no-name with AMD-processor, Win10 and a completey different Antivirus-vendor)
thx + BR / Mats

[fth0]

Regarding the VirtualBox hardening log, there could be some adversary that VirtualBox doesn't recognize as such, and software like Cortex XDR Endpoint Protection would certainly be a candidate.

Inside the Windows Task Manager, take a look at the Processes tab and sort the table by the Disk I/O column, to see which process(es) are responsible.

Generally, you could try and activate using the Host I/O Cache in the VM settings. And you could provide 2 vCPUs to the VM. See, if one or both of those settings influence the behavior.

Note that a term like slow is wide open to interpretation. If you like, you can share some SysBench results with accompanying information about the (physical or virtual) CPU and disk drive. I also happen to use SysBench sometimes.

[MatsBoekelund]

Hi fth0, thanks.
It appears it's the VirtualBoxManager that's doing the heavy disk reading during the powerdown process.

As for giving the Guest one more CPU: I usually configure them with two CPUs, the single CPU in the current one was just a trial to see if I was choking the host. I've never had an issue using 2x CPU in the Guests, also not with the Dell 7280, but I figured it was worth a try. The Dell 7280 has an i5-6200U CPU which is said to have two cores / four threads.

As for enabling Host I/O Cache, it is enabled according to the VBoxManager GUI, on both the IDE- and the SATA controller. Currently the .vdi hangs on the IDE controller rather than on the SATA controller, that was also just another parameter I tried changing. Back when mainly installing the Guests with CentOS 6.2 rather than CentOS 7.2, having the .vdi on a SATA controller used to slow things down considerably. (I know it's supposed to be the other way around but this was repeatable with CentOS 6.2, move the disk(s) to the IDE controller and they performed much better, past experience with CentOS 7.2 Guest has been that also SATA works fine)

Just for kicks I disabled Host I/O cache for the IDE controller in the GUI, things turned even slower. The powerdown process was at least three times as long, the usage was (perceived to be, not measured;-) slightly slower than before but surely not changed by a factor of three like the powerdown. Probably two separate issues.... After turning it back on using vboxmanage. Looking for settings with "I/O", ai also tried --apic off and ioapic off but noticed no change. (in retrospect I now remember having played with Host I/O cache before...)

Next I will import a completely different VM, made by someone else, and see how that plays.
thx + BR / Mats

[MatsBoekelund]

An update: Turns out I should have first tried out someone else's VM quite a while ago, that seems to resolve the most important of my issues, namely the sluggish performance when using the VM. As time permits, later on I will drill down and figure out what the culprit is for that issue. The delayed powerdown is annoying but not a show-stopper like the performance issue. With a couple of days vacation coming up I won't be revisiting this until earliest Wednesday of next week. BR / Mats

[fth0]

As for enabling Host I/O Cache, it is enabled

My bad, I should have noticed it in the VBox.log file.

With a couple of days vacation coming up

Have a nice vacation!