Trouble With 3D Acceleration In Windows XP Guest ("Unable to set graphics mode")

Discussions about using Windows guests in VirtualBox.
Post Reply
BenTheVirtualizer
Posts: 5
Joined: 24. Jul 2021, 08:53

Trouble With 3D Acceleration In Windows XP Guest ("Unable to set graphics mode")

Post by BenTheVirtualizer »

Hi everyone,

I have a computer game requiring 3D acceleration that I want to run in a Windows XP virtual machine, but I’m having no luck. When I run the program, the screen goes black for a second like it’s about to start running the game, but then I get an error message saying “Unable to set graphics mode”. I don’t know what to do about this. It actually does work fine on the host system (Windows 10), but I’d also like to run it on the virtual machine if possible as well. Here’s what I’ve tried:

On VirtualBox 5.2.14, I’ve installed the VirtualBox Guest Additions, using safe mode to install Direct 3D, and I’ve made sure 3D acceleration and 2D video acceleration are enabled in the VirtualBox display settings of the virtual machine. However, I still get the “Unable to set graphics mode” error each time I try to run the program. For good measure, I’ve tried running it in Normal Startup mode, running it in safe mode, running it while both the host and guest operating systems were set to a low resolution, running it as administrator, and running it under the restrictive display settings under “Properties/Compatibility” (which is obtained by right clicking on the program). I’ve tried selecting both 32-bit and 64-bit Windows XP in VirtualBox when setting up the virtual machine and doing these things under both. (The XP system itself is 32-bit, to be clear.) Nothing has worked. I’ve also tried quite a few combinations of the above, but maybe the answer is a combination I haven’t tried?

I also tried using VirtualBox 6.0.24 – my understanding is that 3D acceleration support for Windows XP was discontinued for 6.1.x but that 6.0.x had various options for graphics controllers, so I figured it was worth a try. However, I made even less progress here – even after installing the Guest Additions with Direct 3D and enabling 3D acceleration, I got an error message saying “Unable to create 3D Scene”. I tried using each graphics controller offered by VirtualBox: VboxVGA, VboxSVGA, and VMSVGA, but got the same error message each time. (In contrast, VirtualBox 5.2.14 only gives this error message when 3D acceleration is disabled.)

Some other information: my host operating system is 64-bit Windows 10, and like I said, the guest operating system is 32-bit Windows XP. I have 243 GB available to the host and 1.95 GB available to the guest. I've also attached VBoxHardening.log.

It may also be worth noting that I was unable to correctly handle the d3d8.dll and d3d9.dll files, and (possibly as a result) the DirectX 7 part of the Direct X diagnostic test failed. Section 12.3.12 of the VirtualBox manual says this probably isn’t an issue for 3D acceleration in Windows XP guests, but I thought it was worth noting in case any of you think this actually is worth looking into further.

Many thanks in advance for your help. I know VirtualBox isn’t exactly designed for computer games, but I know others have been able to have at least some success, so I’m hoping there’s a way forward – at least, I figured it was worth a try. Feel free to tell me if I need to provide even more information.
Attachments
VBoxHardening.zip
(24.45 KiB) Downloaded 30 times
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Trouble With 3D Acceleration In Windows XP Guest ("Unable to set graphics mode")

Post by scottgus1 »

Hardening exit code is 0, so no hardening troubles.

Please Start the VM from full normal shutdown, not save-state. Run until you see the problem happen, then shut down the VM from within the VM's OS if possible. If not possible, close the Virtualbox window for the VM with the Power Off option set.

Right-click the VM in the main Virtualbox window's VM list, choose Show Log. Save the far left tab's log, zip it, and post the zip file, using the forum's Upload Attachment tab.
BenTheVirtualizer
Posts: 5
Joined: 24. Jul 2021, 08:53

Re: Trouble With 3D Acceleration In Windows XP Guest ("Unable to set graphics mode")

Post by BenTheVirtualizer »

Thanks for taking a look and for your clear instructions. I was able to follow your instructions, including shutting down the VM from within the VM's OS. I've attached the VBox.log file here.
Attachments
VBox.zip
(26.47 KiB) Downloaded 29 times
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Trouble With 3D Acceleration In Windows XP Guest ("Unable to set graphics mode")

Post by mpack »

BenTheVirtualizer wrote:However, I still get the “Unable to set graphics mode” error each time I try to run the program.
What "graphics mode" is it trying to set?

3D acceleration is a separate issue to availability of graphic modes. If, for example, it was trying to switch to a mode that was incompatible with the host mode (there being only one actual monitor), then the VM can't do the switch while still doing 3D graphics acceleration.
BenTheVirtualizer
Posts: 5
Joined: 24. Jul 2021, 08:53

Re: Trouble With 3D Acceleration In Windows XP Guest ("Unable to set graphics mode")

Post by BenTheVirtualizer »

I don't understand your question. The game has default graphics settings of 800x600, 16 Bit, and Full Screen. If that's not what you're asking for, then what exactly are you asking for?

(And for what it's worth, remember that it is possible to get this program to work on my host system; but like you said, there is only one monitor, so I guess that could be a source of error.)
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Trouble With 3D Acceleration In Windows XP Guest ("Unable to set graphics mode")

Post by mpack »

BenTheVirtualizer wrote:I don't understand your question. The game has default graphics settings of 800x600, 16 Bit, and Full Screen.
Why are you acting pouty like you gave this information already? As far as I can see you did not.

That info is exactly what I'm asking for, and yes I would not expect that the VM can give you 3D acceleration in a 16bit color mode. A VM is just another window in the host graphics system. So hardware accelerated graphics requires co-operation with the host. I would not expect you to be able to get accelerated graphics in a completely incompatible graphics mode. Note that the window size is not a problem, only the color depth is, and the fact that it doesn't match the native color depth.

The host has no problem switching modes since it is not required to maintain compatibility with anything.

In all likelihood your host uses 32bit color. If the game supports that mode then you may have better luck.
BenTheVirtualizer
Posts: 5
Joined: 24. Jul 2021, 08:53

Re: Trouble With 3D Acceleration In Windows XP Guest ("Unable to set graphics mode")

Post by BenTheVirtualizer »

First of all, I'm sorry for coming across as pouty. I genuinely was unclear on whether I was giving the information you were asking for or whether I was understanding your question incorrectly, but rereading the post, I realize that it actually does come across a bit rudely, so I apologize.

On the issue of color modes, yes, my host does have 32-bit color. The game actually does have a 32-bit option (as well as a 24-bit option for whatever it's worth), but even after running it in 32-bit mode, I still got the "Unable to set graphics mode" error, so I'm not sure where to go from here. I looked into reducing the host's bit color, but I couldn't find a way to do that on Windows 10 (And right clicking on VirtualBox and going into Properties and Compatibility shows that the Reduced Color Mode option is greyed out), so I'm at a bit of a loss. I was also wondering if changing the color settings within the guest might somehow help, but I had no luck there either (unless there was some necessary combination of settings that I missed). Do you have any other ideas? Thanks for your help, and sorry again for the earlier miscommunication.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Trouble With 3D Acceleration In Windows XP Guest ("Unable to set graphics mode")

Post by mpack »

BenTheVirtualizer wrote:I looked into reducing the host's bit color
You definitely don't want to do that. Modern Windows and apps do not expect '90s era color capabilities.

I don't see much else I can do to help you here. Hardware acceleration can't be emulated, 'cause then it isn't hardware accelerated any more. So if the hardware accelerator refuses a particular video mode then that's it. Your only option is to abandon ship or disable hardware acceleration, and see if performance is acceptable without it.
BenTheVirtualizer
Posts: 5
Joined: 24. Jul 2021, 08:53

Re: Trouble With 3D Acceleration In Windows XP Guest ("Unable to set graphics mode")

Post by BenTheVirtualizer »

If 3D acceleration is disabled, I just get a "Unable to set 3D Scene" error, so I guess it's probably a lost cause. Like I said, I understand VirtualBox itself wasn't designed for gaming, but I guess it was worth a try. Thanks for your help!
Post Reply