[Cause + Workaround] Graphics freeze on boot with Windows guests + Guest Additions (#16843)

Discussions related to using VirtualBox on Linux hosts.
Post Reply
Posts: 5
Joined: 17. Jun 2017, 05:16

[Cause + Workaround] Graphics freeze on boot with Windows guests + Guest Additions (#16843)

Post by machfour »

Hi all,

I have been having this issue for around the past year, even as I have stayed up to date with the latest Virtualbox releases.
After guest additions are installed, my windows VMs do not boot properly. The Windows splash screen freezes during boot up and there is nothing I can do except power off the machine.

I am using Arch Linux on an IvyBridge CPU with Intel HD graphics 4000 (i915), and have reproduced the issue under the following conditions:

- both windows 8.1 and windows 10
- both intel's xf86-video-intel driver as well as the modesetting driver
- both the stock Arch linux binary package and Oracle's official package (virtualbox-bin from the AUR)
- the linux-ck-ivybridge kernel (4.11.5-2-ck-ivybridge, which I normally use) as well as the stock Arch Linux kernel (4.11.5-1-ARCH)

I think this issue is related to the one called 'Get frozen screen when boot after extension pack installation' (t=83169) (sorry can't link directly as this is my first post).
However I don't observe the problem on a fresh Windows install without Guest Additions installed, which is not mentioned in that post.

I have attached zips of a clean boot/shutdown cycle when the problem happens, for both OSE and non-OSE builds of Virtualbox, as well as the .vbox file for my Windows 10 VM.

Let me know if there's any other information I should provide, or if I should file a bug report.


Edit: There are some similar sounding issues floating around, but I think this snippet of the log stands out as different from anything else I can find, except for that one forum post I referred to above.
00:00:22.665667 VMMDev: Guest Log: VBoxMP::vboxVidPnTargetModeSetFromArray: WARNING! :pfnAddMode failed, Status 0xc01e030b
00:00:22.666096 VMMDev: Guest Log: TargetMode: ID: 1, VSI: VStd(D3DKMDT_VSS_OTHER), totSize(293X68), activeSize(293X68), VSynch(-2/-2=1), PixelRate(-2), ScanLineOrdering(D3DDDI_VSSLO_PROGRESSIVE), Preference(D3DKMDT_MP_NOTPREFERRED)
00:00:22.666158 VMMDev: Guest Log: VBoxMP::vboxVidPnApplyInfoForPathTarget: WARNING! :vboxVidPnTargetModeSetFromArray failed Status(0xc01e030b)
00:00:22.666171 VMMDev: Guest Log: VidPn: ---------
00:00:22.666185 VMMDev: Guest Log:  >>**** Start Dump VidPn Path ****>>
00:00:22.666216 VMMDev: Guest Log: VidPnSourceId(0),  VidPnTargetId(0)
00:00:22.666228 VMMDev: Guest Log: Source NOT Pinned
00:00:22.666239 VMMDev: Guest Log: Target NOT Pinned
00:00:22.666502 VMMDev: Guest Log:   --Transformation: Scaling(D3DKMDT_VPPS_UNPINNED), ScalingSupport(1), Rotation(D3DKMDT_VPPR_UNPINNED), 
        RotationSupport(17)--Importance(D3DKMDT_VPPI_PRIMARY), TargetColorBasis(D3DKMDT_CB_UNINITIALIZED), Content(D3DKMDT_VPPC_NOTSPECIFIED), 
        VFA_TL_O(0X0), VFA_BR_O(0X0), CCDynamicRanges: FirstChannel(0), SecondChannel(0), ThirdChannel(0), FourthChannel(0)| 
        CProtection: Type(D3DKMDT_VPPMT_NOPROTECTION), TODO| GammaRamp: Type(D3DDDI_GAMMARAMP_DEFAULT), DataSize(0), TODO: dump the rest
00:00:22.666521 VMMDev: Guest Log:  <<**** Stop Dump VidPn Path ****<<
00:00:22.666536 VMMDev: Guest Log:   >>>+++SourceMode Set for Source(0)+++
00:00:22.666567 VMMDev: Guest Log:   <<<+++End Of SourceMode Set for Source(0)+++
00:00:22.666582 VMMDev: Guest Log:   >>>---TargetMode Set for Target(0)---
00:00:22.666612 VMMDev: Guest Log:   <<<---End Of TargetMode Set for Target(0)---
00:00:22.666616 VMMDev: Guest Log: ------
00:00:22.666626 VMMDev: Guest Log: MonModeSet: --------
00:00:22.666646 VMMDev: Guest Log:  Tgt[0]
00:00:22.667104 VMMDev: Guest Log: MonitorMode: ID: 1025, VSI: VStd(D3DKMDT_VSS_OTHER), totSize(640X480), activeSize(640X480), VSynch(-2/-2=1), 
        PixelRate(-2), ScanLineOrdering(D3DDDI_VSSLO_PROGRESSIVE), ColorBasis: D3DKMDT_CB_SRGB, Ranges: FirstChannel(8), SecondChannel(8), 
        ThirdChannel(8), FourthChannel(0), MonCapOr: D3DKMDT_MCO_DRIVER, Preference(D3DKMDT_MP_NOTPREFERRED)
[other, similar monitor modes with different resolutions]
00:00:22.676182 VMMDev: Guest Log: ------
00:00:22.676227 VMMDev: Guest Log: VBoxMP::VBoxVidPnCofuncModality: WARNING! :vboxVidPnApplyInfoForPathTarget failed Status(0xc01e030b
00:00:22.676350 VMMDev: Guest Log: Modality Info: PivotType(D3DKMDT_EPT_NOPIVOT), SourceId(0xffffffff), TargetId(0xffffffff),
00:00:22.676438 VMMDev: Guest Log: VBoxMP::DxgkDdiEnumVidPnCofuncModality: WARNING! :VBoxVidPnCofuncModality failed Status(0xc01e030b)
(32.1 KiB) Downloaded 58 times
(27.33 KiB) Downloaded 55 times
(2.14 KiB) Downloaded 56 times
Last edited by socratis on 23. Jun 2017, 14:30, edited 2 times in total.
Reason: Added ticket to the title.
Posts: 5
Joined: 17. Jun 2017, 05:16

Re: Graphics freeze on boot with Windows guests + Guest Additions

Post by machfour »

After some searching, I found that the error code 0xc01e030b returned by pfnAddMode corresponds (according to this) to STATUS_GRAPHICS_INVALID_ACTIVE_REGION, meaning that the active region supplied in pMonitorSourceModeInfo was invalid.

Looking at the active size of 293x68 given there, it's not really that surprising that it's invalid, it's such a small area of screen. So I am wondering, where does this size get set in the first place?

I have been trying to look through the VirtualBox WDDM source code, but it is a struggle. I think that the relevant/buggy magic happens in VBoxVidPnCofuncModality in VBoxMPVidPn.cpp, but it's hard for me to figure out exactly what is going on since I have never done graphics programming before.
Posts: 5
Joined: 17. Jun 2017, 05:16

Re: Graphics freeze on boot with Windows guests + Guest Additions

Post by machfour »

After even more digging, I believe I have found the cause of this bug.

As I mentioned before, I am using i3 as my window manager on Linux. I have Virtualbox windows set to be floating, not tiled, but I think I am affected by some variant of the tiling WM resize bug #15863.
Therefore, when I start up my VMs using the QT GUI, they are automatically resized to be a very small window size (say ~300x100) at several points during the guest VM boot sequence.

Somewhere along the line, this resolution is saved by Virtualbox as a 'custom display resolution', which then the display drivers seem to try and restore on each boot.

Without Guest Additions installed, this is fine, but just annoying to have to manually resize the display every time the VM is booted. However when the WDDM driver is installed, the VidPn structures in Windows kick in, and try to add this mode, which can't be handled (pfnAddMode fails), and so the whole thing borks every time the VM is booted after Guest Additions are installed.

What is the mechanism for this? Well, in the (guest) registry, this overly-small resolution is saved by the WDDM driver as a 'CustomXRes' and 'CustomYRes' (see the picture attached). These values are then loaded at boot time, and seem to be prioritised over more sensible resolutions. I noticed that the relevant-looking function VBoxMPValidateVideoModeParamsGuest in src/VBox/Additions/WINNT/Graphics/Video/mp/common/VBoxMPVidModes.cpp, *does not actually check the resolution* of the video mode it is given to validate. So I think this is possibly the defect here, but I am not 100% sure.

In the end I got a VM to boot after installing Guest Additions (finally, after about a year of this problem!) with the following workaround.
Note that I needed to start with NOT having the Guest Additions installed, so that I could boot and access the registry after installing the WDDM driver. In other words, if your only VM has this problem and won't boot, I'm not sure how to get around this. Unless there is some way of editing the registry without booting the VM.
The necessary step is to edit the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\4d36e968-e325-11ce-bfc1-08002be10318\Properties\Custom{X,Y}Res keys, and set them to something more sensible, like 800 and 600 respectively. Again, see the picture below.

I will file have filed a bug (#16843) against the WDDM driver about this. But, I can see why this bug is not a common one, since the preconditions are only created by having a tiling window manager. (Although once I had run the VM in i3, even starting it in another non-tiling one such as Xfce did not fix the problem, since the tiny custom resolution had already been saved in the registry).

If anyone knows more about the custom resolution behaviour, or how to edit the registry without starting the VM, please post here :)

EDIT: I have attempted to edit the registry offline both in another VM (by mounting the VDI inside it) and also with the recovery CD. Neither method works as HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet does not appear and the ControlSet001 tree does not have a the corresponding Property described above.

Screenshot of registry key showing overly small custom X and Y resolutions:
regedit.png (122.2 KiB) Viewed 3123 times
Posts: 1
Joined: 9. Nov 2017, 16:34

Re: [Cause + Workaround] Graphics freeze on boot with Windows guests + Guest Additions (#16843)

Post by karstenkanon »

Thanks for finding this machfour!

I can confirm that this issue is happening - I'm on a (Arch) Linux host systen, using Gnome and the ShellShape gnome shell extension (i.e. a partly tiling window manager). Mucked around with this issue for a while before finding this post - followed the instructions (i.e. booting to safe mode and going to the registry).

For whatever reason, I didn't have permissions to go to the full path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\4d36e968-e325-11ce-bfc1-08002be10318\Properties\ - it complained when I attempted to go to "Properties" - but the CustomXRes and CustomYRes keys were available when marking HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\4d36e968-e325-11ce-bfc1-08002be10318 - and editing them there worked just fine.
Posts: 5
Joined: 14. Nov 2017, 11:14

Re: [Cause + Workaround] Graphics freeze on boot with Windows guests + Guest Additions (#16843)

Post by dgrigorev »

Hello Karstenkanon.
The problem is in Guest Additions, so host type is irrelevant here.
Recent 5.2 GA should have appropriate fix.
Please upgrade GA in your Windows guest and if the problem will persist - attach VBox.log here.
Post Reply