VM Screen resizing grayed out, vboxvideo.ko not loading, 3D acceleration stalls VM

[Enquirer]

Hello,

I did a chain upgrade of my Fedora 21 VM: 21->22->23->24->25->26->27. After the last upgrade to 27, the VM booted into black screen. I was running VB 5.2.10 on MacOS 10.13

Things I tried:

• replaced sddm with kdm: got to login screen, but window manager could not render any objects. Display resolution controls are greyed out and the resolution is low.
• disabled 3D Acceleration: everything works fine, but display resolution controls are greyed out and the resolution is low
• noticed that vboxvideo module was not loaded. Manual loading failed with:

  Code: Select all

  modprobe: ERROR: could not insert 'vboxvideo': Invalid argument

• attempted to rebuild guest additions multiple times, doing removals, reboots, clean installations, replacements etc. All to no avail. The build log always has the following:

  Code: Select all

    gcc -Wp,-MD,/tmp/vbox.0/.vboxvideo.mod.o.d  -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/8/include -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -DCONFIG_CC_STACKPROTECTOR -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -fno-PIE -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_AVX512=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mindirect-branch=thunk-extern -mindirect-branch-register -DRETPOLINE -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -DCC_HAVE_ASM_GOTO -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fvar-tracking-assignments -g -pg -mfentry -DCC_USING_FENTRY -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -Wno-packed-not-aligned -I/lib/modules/4.16.9-300.fc28.x86_64/build/include -I/tmp/vbox.0 -Iinclude/drm -D__KERNEL__ -DMODULE  -DKBUILD_BASENAME='"vboxvideo.mod"'  -DKBUILD_MODNAME='"vboxvideo"' -DMODULE  -c -o /tmp/vbox.0/vboxvideo.mod.o /tmp/vbox.0/vboxvideo.mod.c
    ld -r -m elf_x86_64 -z max-page-size=0x200000 -T ./scripts/module-common.lds --build-id  -o /tmp/vbox.0/vboxvideo.ko /tmp/vbox.0/vboxvideo.o /tmp/vbox.0/vboxvideo.mod.o ;  true
  make V=1 CONFIG_MODULE_SIG= -C /lib/modules/4.16.9-300.fc28.x86_64/build SUBDIRS=/tmp/vbox.0 SRCROOT=/tmp/vbox.0 INSTALL_MOD_PATH= INSTALL_MOD_DIR=misc modules_install
  test -e include/generated/autoconf.h -a -e include/config/auto.conf || (		\
  echo >&2;							\
  echo >&2 "  ERROR: Kernel configuration is invalid.";		\
  echo >&2 "         include/generated/autoconf.h or include/config/auto.conf are missing.";\
  echo >&2 "         Run 'make oldconfig && make prepare' on kernel src to fix it.";	\
  echo >&2 ;							\
  /bin/false)
  mkdir -p /lib/modules/4.16.9-300.fc28.x86_64/misc
  make -f ./scripts/Makefile.modinst
    mkdir -p /lib/modules/4.16.9-300.fc28.x86_64/misc ; cp /tmp/vbox.0/vboxvideo.ko /lib/modules/4.16.9-300.fc28.x86_64/misc ; true /lib/modules/4.16.9-300.fc28.x86_64/misc/vboxvideo.ko ; scripts/sign-file "sha256" "certs/signing_key.pem" certs/signing_key.x509 /lib/modules/4.16.9-300.fc28.x86_64/misc/vboxvideo.ko || true && true /lib/modules/4.16.9-300.fc28.x86_64/misc/vboxvideo.ko
  At main.c:160:
  - SSL error:02001002:system library:fopen:No such file or directory: crypto/bio/bss_file.c:74
  - SSL error:2006D080:BIO routines:BIO_new_file:no such file: crypto/bio/bss_file.c:81
  sign-file: certs/signing_key.pem: No such file or directory
    /bin/sh ./scripts/depmod.sh /sbin/depmod 4.16.9-300.fc28.x86_64 ""
  Creating user for the Guest Additions.
  Creating udev rule for the Guest Additions kernel module.

  Not sure if this SSL error is a red herring or the real cause. The stock Fedora kernel seems to be configured to not enforce module signing and to automatically sign modules it loads:

  Code: Select all

  grep _SIG_ /boot/config-4.16.9-300.fc28.x86_64
  # CONFIG_MODULE_SIG_FORCE is not set
  CONFIG_MODULE_SIG_ALL=y
  # CONFIG_MODULE_SIG_SHA1 is not set
  # CONFIG_MODULE_SIG_SHA224 is not set
  CONFIG_MODULE_SIG_SHA256=y
  # CONFIG_MODULE_SIG_SHA384 is not set
  # CONFIG_MODULE_SIG_SHA512 is not set
  CONFIG_MODULE_SIG_HASH="sha256"
  CONFIG_KEXEC_SIG_FORCE=y
  CONFIG_MODULE_SIG_KEY="certs/signing_key.pem"
  

• upgraded VB to 5.2.12: same issue
• upgraded guest OS to Fedora 28: same issue

I still have the original VM with Fedora 21 running just fine on 5.2.10 and 5.2.12 with 3d acceleration, vboxvideo.ko loaded and guest display resolution controls enabled.

Any suggestion on how to troubleshoot and fix would be much appreciated.

[socratis]

upgraded guest OS to Fedora 28: same issue

Can you try with a fresh install instead of an upgrade? I don't recall any issues with a stock Fedora 28 installation and VirtualBox 5.2.12...

[Enquirer]

I could try testing fresh Fedora 28 install as a proof of concept. However, I do want to make my existing VM work. Moreover, I am puzzled by the problem with VB that surfaced with Fedora 27 and is now in F-28 and would like to understand how to troubleshoot and fix.

[socratis]

How do you know it's a VirtualBox problem and not a F21->22->23...28 problem? I could see some potential problems if I were to update my WinNT 3.51 workstation all the way to Win10...

That's why I want you to try Fedora 28. I've installed it 4-5 times now, and besides some minor glitches (before 5.2.12) I haven't seen anything that would break it...

[Enquirer]

That's why I want you to try Fedora 28. I've installed it 4-5 times now, and besides some minor glitches (before 5.2.12) I haven't seen anything that would break it...

Installed Fedora 28 Workstation with KDE. It worked out-of-the-box with the GA shipped with the distro. Window resizing worked fine, etc.
However, when I upgraded to GA 5.2.12, things started to break: everything became sluggish, resizing has several seconds lag and parts of the display resize "asynchronously". Most importantly (from the usability standpoint), the information sent to the Window Manager about the window size must be incorrect, because the panel at the bottom (with the start button, etc) is off the screen and cannot be accessed (I can see it for a brief moment when KDE starts or when I resize the window, and then it moves down off the screen). Resizing the screen a few time in a row "hangs" the refresh process of the VB so that in interesting effect can be seen: the VM window can be moved, while its content does not move (see attached image).

Screen Shot 2018-05-25 at 11.01.57 AM.png (123.73 KiB) Viewed 4739 times

Perhaps a problem with the VB version for MAC?

[socratis]

Yes, a known issue with 4 possible workarounds. See #17168: VB display frame and content are no longer bound in High Sierra (osx 10.13).

As for the sluggish behavior, haven't observed it myself. Maybe a 10.13.x issue? I'm on 10.11.6 on a MacBookPro11,5.

[Enquirer]

Turning off 3D acceleration on the newly installed F28 helped with sluggishness and frame and content bonding.

I might have solved the issue I was having with the VM I upgraded to F28:

• Removed GA I downloaded from Oracle and installed the GA rpm package that was part of F28
• Removed 'nomodeset' from the kernel parameters

While the first step allowed to normally boot into KDE with 3D acceleration enabled, it did not allow window resizing (vboxvideo was still not loading). Removing the 'nomodeset' from the kernel after that, makes vboxvideo load automatically and window resizing is now working.

[socratis]

Removed GA I downloaded from Oracle and installed the GA rpm package that was part of F28

I don't particularly like that part, it might break other things, like shared folders, drag and drop, etc.

Removed 'nomodeset' from the kernel parameters

That was needed because the graphics drivers have now moved to the kernel. That's a side effect of upgrading, instead of installing F28 straight.

allowed to normally boot into KDE with 3D acceleration enabled

That's another key difference, I've been trying with the default Gnome one, not the KDE one. I might download the KDE flavor and try it out...