Can't get graphics hardware acceleration to work on a Gentoo Guest

Discussions about using Linux guests in VirtualBox.
Post Reply
aisix
Posts: 4
Joined: 27. Jul 2022, 14:54

Can't get graphics hardware acceleration to work on a Gentoo Guest

Post by aisix »

The system
Version of VirtualBox installed: Virtualbox 6.1.36r152435 (Qt5 6.2)
Type and version of the Host Operating System: Windows 7 SP1
Type and version of the Guest OS: Gentoo GNU/Linux x86_64

Relevant host system information:
  • DirectX Version: DirectX 11
  • Card name: NVIDIA GeForce GT 240
  • Driver Name: nvd3dumx.dll,nvwgf2umx.dll,nvwgf2umx.dll,nvd3dum,nvwgf2um,nvwgf2um
  • OpenGL Extensions Viewer reports support for OpenGL 3.3.0
This will become relevant when reading the virtualbox log (attached as file as the forum posting guidelines request and compressed to fit in the 128kb filesize limit).
Gentbox-2022-08-06-09-25-01.zip
virtualbox log
(32.12 KiB) Downloaded 8 times
In particular, these lines

Code: Select all

00:00:04.966061 supR3HardenedErrorV: supR3HardenedScreenImage/LdrLoadDll: rc=VERR_CR_PKCS7_KEY_USAGE_MISMATCH fImage=1 fProtect=0x0 fAccess=0x0 \Device\HarddiskVolume3\Windows\System32\nvd3dumx.dll: WinVerifyTrust failed with hrc=CERT_E_CHAINING on '\Device\HarddiskVolume3\Windows\System32\nvd3dumx.dll'
00:00:04.966144 supR3HardenedErrorV: supR3HardenedMonitor_LdrLoadDll: rejecting 'C:\Windows\system32\nvd3dumx.dll': rcNt=0xc0000190
00:00:04.966851 supR3HardenedErrorV: supR3HardenedScreenImage/NtCreateSection: cached rc=VERR_CR_PKCS7_KEY_USAGE_MISMATCH fImage=1 fProtect=0x2 fAccess=0x7 cHits=1 \Device\HarddiskVolume3\Windows\System32\nvd3dumx.dll
00:00:04.968026 supR3HardenedErrorV: supR3HardenedScreenImage/NtCreateSection: cached rc=VERR_CR_PKCS7_KEY_USAGE_MISMATCH fImage=1 fProtect=0x2 fAccess=0x7 cHits=2 \Device\HarddiskVolume3\Windows\System32\nvd3dumx.dll
The problem
First of all, I want to acknowledge that Windows 7 is not in the list of supported Host systems of the user manual, but I'll ask anyways in hopes that my problem can be looked over to check for any errors unrelated to that.
I am trying to have hardware acceleration on my guest system, but the system reports that software acceleration is being used instead.

Code: Select all

glxinfo -B
...
    Device: llvmpipe (LLVM 14.0.6, 128 bits) (0xffffffff)
    Accelerated: no
...
which, according to the MESA documentation:
The Gallium llvmpipe driver is a software rasterizer that uses LLVM to do runtime code generation.
Relevant lines from Xorg.log.0

Code: Select all

[  4470.756] (II) Loading sub module "dri2"
[  4470.756] (II) LoadModule: "dri2"
[  4470.756] (II) Module "dri2" already built-in
[  4470.756] (II) Loading sub module "dri3"
[  4470.756] (II) LoadModule: "dri3"
[  4470.756] (II) Module "dri3" already built-in
[  4470.756] (II) vmware(0): Initialized VMWARE_CTRL extension version 0.2
[  4470.758] (WW) vmware(0): Failed to initialize Gallium3D Xa. No render acceleration available.
[  4470.758] (WW) vmware(0): Skipped initialization of direct rendering due to lack of render acceleration.
[  4470.758] (--) vmware(0): Render acceleration is disabled.
[  4470.758] (==) vmware(0): Rendercheck mode is disabled.
[  4470.758] (--) vmware(0): Direct rendering (DRI2 3D) is disabled.
[  4470.758] (--) vmware(0): Direct rendering (DRI3 3D) is disabled.
[  4470.759] (II) vmware(0): No 3D acceleration. Not setting up textured video.
[  4470.759] (WW) vmware(0): Disabling Xv because no adaptors could be initialized.
[  4470.760] (II) Initializing extension GLX
[  4470.760] (II) AIGLX: Screen 0 is not DRI2 capable
[  4470.767] (II) IGLX: Loaded and initialized swrast
[  4470.767] (II) GLX: Initialized DRISWRAST GL provider for screen 0
relevant lines from dmesg

Code: Select all

[    6.532652] [TTM] Zone  kernel: Available graphics memory: 1532462 KiB
[    6.532772] vmwgfx 0000:00:02.0: [drm] FIFO at 0x00000000f0000000 size is 2048 kiB
[    6.532871] vmwgfx 0000:00:02.0: [drm] VRAM at 0x00000000e0000000 size is 131072 kiB
[    6.532883] vmwgfx 0000:00:02.0: [drm] Running on SVGA version 2.
[    6.532889] vmwgfx 0000:00:02.0: [drm] DMA map mode: Caching DMA mappings.
[    6.532945] vmwgfx 0000:00:02.0: [drm] Legacy memory limits: VRAM = 131072 kB, FIFO = 2048 kB, surface = 393216 kB
[    6.532949] vmwgfx 0000:00:02.0: [drm] MOB limits: max mob size = 0 kB, max mob pages = 0
[    6.532954] vmwgfx 0000:00:02.0: [drm] Capabilities: cursor, cursor bypass 2, alpha cursor, 3D, extended fifo, pitchlock, irq mask, gmr, traces, gmr2, screen object 2, 
[    6.532958] vmwgfx 0000:00:02.0: [drm] Max GMR ids is 8192
[    6.532961] vmwgfx 0000:00:02.0: [drm] Max number of GMR pages is 1048576
[    6.532964] vmwgfx 0000:00:02.0: [drm] Maximum display memory size is 131072 kiB
[    6.535800] vmwgfx 0000:00:02.0: [drm] Screen Object display unit initialized
[    6.536255] vmwgfx 0000:00:02.0: [drm] Fifo max 0x00200000 min 0x00001000 cap 0x00000355
[    6.536268] vmwgfx 0000:00:02.0: [drm] Available shader model: Legacy.
[    6.536274] [drm:vmw_host_printf [vmwgfx]] *ERROR* Failed to send host log message.
[    6.567618] fbcon: svgadrmfb (fb0) is primary device
[    6.568525] Console: switching to colour frame buffer device 100x37
[    6.570094] [drm] Initialized vmwgfx 2.19.0 20210722 for 0000:00:02.0 on minor 0
[   11.159773] 11:46:40.714613 main     VBoxService 6.1.36_Gentoo r152435 (verbosity: 0) linux.amd64 (Aug  6 2022 06:08:05) release log
               11:46:
[   11.160247] 11:46:40.715991 main     OS Product: Linux
[   11.161028] 11:46:40.716421 main     OS Release: 5.15.52-gentoo
[   11.162422] 11:46:40.717263 main     OS Version: #8 SMP Wed Jul 27 10:43:34 -03 2022
[   11.165050] 11:46:40.718756 main     Executable: /usr/sbin/vboxguest-service
               11:46:40.718763 main     Process ID: 1771
               11:46:40.718769 main
[   11.172993] 11:46:40.727773 main     6.1.36_Gentoo r152435 started. Verbose level = 0
[   11.176114] 11:46:40.730819 main     vbglR3GuestCtrlDetectPeekGetCancelSupport: Supported (#1)
[   11.202685] vboxsf: Unknown parameter 'tag'
[   11.205875] 11:46:40.761189 automount vbsvcAutomounterMountIt: Running outdated vboxsf module without support for the 'tag' option?
[   11.206677] 11:46:40.762085 automount vbsvcAutomounterMountIt: Successfully mounted 'vbox-shared' on '/mnt/vbox-shared'
[   23.127425] Service: Shared Clipboard
[   23.135482] Service: Host Version Check
[   23.157282] Service: Seamless
[   23.163432] Service: Drag and Drop (DnD)
[   23.167109] Service: Display SVGA X11
[   23.177204] Proxy window=8388609, root window=1335 ...
[   23.177400] Started
[   23.177611] 6.1.36_Gentoor152435
After trying everything I could think of on the Guest side of things, my best guess is those lines on the virtualbox log about nvd3dumx are very relevant and must be the key to solving this issue.
The research
I have searched through the user manual, read through chapters 3.6.1. Screen Tab, 4.5.1. Hardware 3D Acceleration (OpenGL and Direct3D 8/9) and 9.3.2. Guest Graphics and Mouse Driver Setup in Depth; as well as the gentoo wiki entry on Virtualbox
From reading that I gathered that I would need to:
  • use VMSVGA as my graphics controller and enable 3d acceleration in the screen tab of the virtual machine settings
  • install the guest additions (from the gentoo native package manager, portage)
  • add the virtualbox-guest-additions and dbus service to the default runlevel
  • have support for virtualbox and it's modules in the kernel
  • add my user to the video, vboxguest and vboxsf groups
  • have the VBoxClient program running as a client in the X server (added VBoxClient-all as a line in my xinitrc, which makes it so the X server doesn't run the first time, but from the second attempt onwards works)
  • build and use the vmware driver for X and MESA since I'm using VMSVGA
  • maybe use a custom xorg.conf as provided in the advanced topics of the user manual (this did not work at all, the X server wouldn't start)
The result of following these steps (except for the last one) yielded working auto-resizing and shared clipboard, but hardware acceleration did not work.

And so, I started searching for virtualbox hardware acceleration issues in this and other forums, and tried many different things listed as solutions to them.
One of the first things I found out for guest-additions not working was the guest additions version number not matching the Host system's virtualbox version number, so I checked and saw that they didn't, the stable version of the virtualbox-guest-additions in the gentoo repo is 6.1.34.
I installed the unstable version which was 6.1.36, but nothing changed other than the versions now match.
Another reason for hardware acceleration not working I found was not giving the guest system enough video memory.
After changing it from 16mb to 128mb hardware acceleration still didn't work.
On that same post (can't post link because I'm a new user), another reason for non-working hardware acceleration was giving only 1 CPU to the guest system.
This should not be the issue since I gave it 3.
A different post said that distribution repository installed guest-additions were known not to work, and that the ones provided in the .iso should be used instead
I uninstalled the package provided by my distribution, mounted the virtualbox provided .iso and tried to install them, which seemed to be succesful but made it so the window size was warped to a smaller size and now the X server wouldn't even start, so I decided to uninstall them and keep using the ones provided by the portage package manager since those at least didn't cause further issues and actually work (besides the hardware acceleration part)
I lost track of where I even read this, but someone said a workaround was setting "nomodeset" on the kernel commandline.
That did not work at all, warped the virtualbox window size and made it so the X server wouldn't start, same as the previous method.
The next reason for non-working hardware acceleration I found was OpenGL symlinks being wrong, but I checked and they point to the right places as stated by the articles that go into this issue.
At this point I gave up and decided I would just ask here and see if I can get better pointers on what I should be looking for or workarounds I haven't tried.
The request
Please help me figure out where the issue is on my setup, it could be lack of corret drivers, lack of support for my host system, my hardware not being good enough, my configuration being wrong or some other thing.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Can't get graphics hardware acceleration to work on a Gentoo Guest

Post by mpack »

It looks like you have a hardening problem with an NVidia DLL. Try updating the host NVidia drivers.
multiOS
Volunteer
Posts: 800
Joined: 14. Sep 2019, 16:51
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: WIN11,10, 7, Linux (various)
Location: United Kingdom

Re: Can't get graphics hardware acceleration to work on a Gentoo Guest

Post by multiOS »

In addition to mpack's advice, you might want to look at replacing the existing ancient Guest Additions installation:
00:00:28.297847 VMMDev: Guest Additions information report: Version 6.0.0 r127566 '6.0.0'
Not only is that version way out of date, but it comes from the first release of the previous and now unsupported 6.0.x version of VirtualBox and there have been many changes since that release.

You should be able to install/update to the matching version of Guest Additions by using the included as an ISO in the 6.1.36 version you have already installed. If using the Virtual GUI then with Gentoo running, use the Devices > Insert Guest Additions CD Image... from the VirtualBox Menu Bar, then navigate to the mounted CD Drive in the File Manager app, launch Terminal and use whatever the Gentoo Command Line to launch the VBoxLinuxAdditions.run file as admin (sudo) . As you already have a version of Guest Additions installed I presume the necessary Linux dependencies have been installed, but please watch the terminal output carefully for any installation errors. The Linux Guest Additions installation procedure is outlined in Section 4.2.2 and the required dependencies are shown in section 2.3.2 if needed.

When the installation completes, please 'eject' the CD/ISO from the File Manager and reboot the VM.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Can't get graphics hardware acceleration to work on a Gentoo Guest

Post by fth0 »

aisix wrote:Please help me figure out where the issue is on my setup
Kudos for your deep analysis. Your problem is a combination of You have two different issues:

1. The VirtualBox error code VERR_CR_PKCS7_KEY_USAGE_MISMATCH indicates a problem with an invalid NVIDIA code signing certificate (for details, see my analysis in Re: 3D accelerated machines crash on reboot). After my analysis, the VirtualBox certificate checks were relaxed a bit in VirtualBox 6.1.24, because there are no security-related implications. This doesn't seem to help in your case.

2. When you installed the VirtualBox Guest Additions (GA), the GA weren't able to build and/or replace the VirtualBox kernel modules inside your guest OS, leading to the distribution's pre-installed 6.0.0 kernel drivers being used.

A successful installation of the GA matching your VirtualBox version should solve your second problem. Pay attention to the log messages in the terminal during the GA installation.

Edit:
I made the mistake of thinking that the GA kernel drivers rejected the NVIDIA driver, but that's already done by the VirtualBox hardening implementation on the host side. Therefore, the version of the GA kernel drivers isn't really important for the main issue.

For posterity, I've edited this post by striking through old text and adding underlined new text.
Last edited by fth0 on 9. Aug 2022, 13:15, edited 1 time in total.
aisix
Posts: 4
Joined: 27. Jul 2022, 14:54

Re: Can't get graphics hardware acceleration to work on a Gentoo Guest

Post by aisix »

Hi, thanks for your help and recommendations.
As you clearly pointed out, the guest additions version being used does not match what I believed to have installed, and after some thinking, I believe it was because of the order of operations I did.
I compiled the kernel with support for the guest additions before even installing them, which must have lead to the modules defaulting to version 6.0 for some reason or another.
I have remedied this by recompiling the kernel with the correct guest additions now installed and the versions now match, but the issue persisted, so on to the next thing.

The graphics card driver.
I have the latest available version of the driver for my Nvidia Geforce GT 240, as listed in nvidia's website.
For thoroughness' sake I uninstalled it and reinstalled it anyways in hopes of the certificate thing getting fixed, but sadly it wasn't.
I tried uninstalling it and, instead of using the one provided in nvidia's website, going to windows' device manager's entry for my graphics card and hitting "update driver".
The rationale behind this decission is what's stated on the thread fth0 linked where nvidia doesn't correctly certify the driver, but a valid microsoft certificate will be accepted in its stead.
This installed a more dated version of the driver (one from 2015 as opposed to the 2016 latest one), which kind of did the trick, but not to my satisfaction.
The aged driver has notably worse performance, which is very undesirable since every drop of performance matters on my computer, and using this driver is worse for every other task that isn't the virtual machine.

I'll reiterate, graphics hardware acceleration is now working, THANK YOU VERY MUCH!!
So don't take this as dissatisfaction with your help, I appreciate it a lot!
But one final thing I would like to know is the following: Is there a way in which I can benefit from the better driver while also having virtualbox work with it? Can I whitelist this driver as "known good" somehow?
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Can't get graphics hardware acceleration to work on a Gentoo Guest

Post by fth0 »

aisix wrote:As you clearly pointed out, the guest additions version being used does not match what I believed to have installed, and after some thinking, I believe it was because of the order of operations I did.
I compiled the kernel with support for the guest additions before even installing them, which must have lead to the modules defaulting to version 6.0 for some reason or another.
I have remedied this by recompiling the kernel with the correct guest additions now installed and the versions now match, but the issue persisted, so on to the next thing.
From your description, I'm not sure if the result is correct. A new (zipped) VBox.log file would prove that. ;)

To describe how it works on another guest OS: When Debian/Ubuntu gets installed inside a VM, it installs a kernel modules package with hundreds of kernel modules, an old version of the GA kernel drivers (e.g. 6.0.0) being amongst them. When you later on install the VirtualBox GA, they try to automatically build their own integrated kernel modules and place them in another location, and the Debian/Ubuntu module system will use them instead of the pre-installed kernel modules.

Note that the GA consist of programs and kernel modules. When the kernel modules couldn't be built (e.g. because no kernel headers were available), the final installation consists of GA 6.1.36 programs and GA 6.0.0 kernel modules. The VBox.log file usually indicates both.

The VirtualBox 6.1.36 vboxguest kernel driver should ignore the certificate problem of the newer NVIDIA driver.

Edit:
I made the mistake of thinking that the GA kernel drivers rejected the NVIDIA driver, but that's already done by the VirtualBox hardening implementation on the host side. Therefore, the version of the GA kernel drivers isn't really important for the main issue.

For posterity, I've edited this post by striking through old text and adding underlined new text.
Last edited by fth0 on 9. Aug 2022, 13:19, edited 1 time in total.
aisix
Posts: 4
Joined: 27. Jul 2022, 14:54

Re: Can't get graphics hardware acceleration to work on a Gentoo Guest

Post by aisix »

I reinstalled the newer driver to check what you said and the issue persists.
I mounted the virtualbox guest additions .iso and ran sh ./VBoxLinuxAdditions.run as the manual states I should do.
Here's the VBox.log with the driver 342.01.
Gentbox-2022-08-08-23-03-10.zip
VBox.log with 342.01 driver
(31.5 KiB) Downloaded 5 times
doing a quick ctrl+f on it, the guest additions version once again seems to match.

Code: Select all

00:00:25.528377 VMMDev: Guest Additions information report: Version 6.1.36 r152435 '6.1.36'
But graphics hardware acceleration does not work, and the output of glxinfo -B and Xorg.log.0 are back to the what the original post had.
I'll post VBox.log with the 341 driver that windows found on it's own after reinstalling it (it takes a bit).
multiOS
Volunteer
Posts: 800
Joined: 14. Sep 2019, 16:51
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: WIN11,10, 7, Linux (various)
Location: United Kingdom

Re: Can't get graphics hardware acceleration to work on a Gentoo Guest

Post by multiOS »

Apparently there are some system changes that need to be applied to Gentoo which are not required by distributions included in VirtualBox's officially supported list. Have you seen Gentoo's own guidance, which includes various trouble shooting guides - https://wiki.gentoo.org/wiki/VirtualBox

Recommending 'Third-Party Guides' isn't usually considered appropriate on this forum, but, as Gentoo isn't based on one of the 'officially supported distributions' and the Gentoo people will know best about their own OS's specific needs....
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Can't get graphics hardware acceleration to work on a Gentoo Guest

Post by fth0 »

Argh! I've been running in the wrong direction :( (and nobody has tried to stop me).

I'll clean up the mess I've created later on ... done.
Last edited by fth0 on 9. Aug 2022, 13:21, edited 1 time in total.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Can't get graphics hardware acceleration to work on a Gentoo Guest

Post by mpack »

multiOS wrote: Recommending 'Third-Party Guides' isn't usually considered appropriate on this forum, but, as Gentoo isn't based on one of the 'officially supported distributions' and the Gentoo people will know best about their own OS's specific needs....
No I'm fine with that. The Gentoo people are clearly the authority on Gentoo, and will indeed very likely know best how to get VirtualBox running well on Gentoo.

The "third party guides" I object to are answers on public question/answer sites where there is no reason to believe that the person answering has any knowledge whatever. In most cases I see they are simply regurgitating stuff they read elsewhere, usually distorted in unfortunate ways, and stripped of all date and place information to tell us when and where that answer might have been correct. Though in many cases you can recognize the wording because it turns out the real authority was us anyway, i.e. some old article in the tutorials section. But the main problem is not that those people are automatically wrong, the main problem is I simply refuse to attempt to parse the correctness of information coming from every tom, dick and blogger across the entire Internet just because someone here references them, nor can we leave bad info unchallenged.
aisix
Posts: 4
Joined: 27. Jul 2022, 14:54

Re: Can't get graphics hardware acceleration to work on a Gentoo Guest

Post by aisix »

For completeness sake, here's the VBox.log using the old driver (reported as 341.44 in the nvidia driver installation program).
Gentbox-2022-08-09-08-39-59.zip
VBox.log with the 341.44 driver
(34.09 KiB) Downloaded 4 times
As I understand it, despite your best efforts to get me to install the guest additions the right way, and despite VBox.log reporting the correct guest additions version is running; I have not managed to do it, or the new driver, despite Nvidia's lack of a valid certificate, should be compatible.
The issue probably arises from differences between gentoo and virtualbox's supported GNU/Linux distributions, so I'll ask for in the gentoo forums.
Thanks for all your help!
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: Can't get graphics hardware acceleration to work on a Gentoo Guest

Post by fth0 »

aisix wrote:Is there a way in which I can benefit from the better driver while also having virtualbox work with it? Can I whitelist this driver as "known good" somehow?
No, the VirtualBox hardening implementation deliberately doesn't allow whitelisting.
aisix wrote:The issue probably arises from differences between gentoo and virtualbox's supported GNU/Linux distributions, so I'll ask for in the gentoo forums.
This won't help IMO. The issue is completely on the Windows host side. The VirtualBoxVM process tries to load the NVIDIA driver, and the VirtualBox hardening implementation denies that. In consequence, the Gentoo guest OS still can talk to its vmwgfx VMSVGA driver, but VirtualBox cannot transport the commands and data to the NVIDIA driver in the Windows host, so this is doomed to fail.

PS: I've corrected my mistakes in my previous posts in this thread.
Post Reply