Kernel module signing not persistent after rebooting Ubuntu host

[milverton]

My aim has been to avoid disabling Secure Boot at the BIOS level merely to run VirtualBox under Ubuntu ( - in this case version 22.04.1 snap-free). Much has been written about this.

I have generated a MOK (machine owner's key) and enrolled it, and that seems to be persistent: it does not need to be repeated.

But I have discovered that the actual signing of the VirtualBox kernel modules does need to be repeated after each reboot of the Ubuntu host, followed by using modprobe to add the driver vboxdrv - both done as root. It doesn't seem to make any difference which kernel version is used. I have found this works with VirtualBox 6.1; and I suspect it also works with VirtualBox 7.06, though I haven't explicitly tested 7.06 with these steps. (fsarchiver enabled the ability to get back from 7.06 to 6.1.)

When I view the properties of the 3 *.ko kernel modules in the folder /lib/modules/$(uname -r)/misc, they all get their timestamps updated each time I do the signing again; and then there's the other step needed, namely to run modprobe vboxdrv. After this, the VMs start up smoothly.

I'm on board with the need to run these steps after a new kernel is installed, but I'm curious if anyone can suggest why they need to be repeated after a straightforward reboot. Depending on the ultimate reason, it seems it might be possible to set things up to avoid the need.

(Added later)
I don't know if this could be a clue to what's going on, but when the machine reboots the timestamps and sizes of the 3 modules take on one (repeatable) set of values, and when the re-signing is run they take on another (repeatable) set of values.

Considering timestamps first, on reboot they set themselves to the reboot time in UTC, but label it as being that same time in the local timezone - in other words, incorrectly. On re-signing, the timestamps align correctly to the time the procedure is run.

File sizes I quote in bytes and in pairs, the number before the slash being after reboot and the number after the slash being after re-signing is run: (I'm guessing it's probably the pattern, or the changes, which have more relevance than the precise vaues themselves.)

vboxnetadp.ko 442,184 / 442,623
vboxnetfit.ko 795,852 / 796,295
vboxdrv.ko 5,601,248 / 5,601,687

[milverton]

I found a page suggesting gcc version could be an issue, with 12 rather than 11 being a solution. I'm running Ubuntu, not Mint. My current gcc version is 11.3.0.

https://forums.linuxmint.com/viewtopic.php?p=2292084

Would anybody care to comment on whether similar steps might be a solution here?

[milverton]

The gcc idea was possibly a red herring in this context. But I might have found where the problem is, and I'd be interested if someone more familiar with shell script would be able to comment on it. There is a file /usr/lib/virtualbox/vboxdrv.sh which seems to be where the 3 kernel module files for vboxdrv get wiped out every time the script runs; and I suspect it runs only on boot. If my hunch is correct on this, it means that signing the kernels in the way I have been doing is destined to need repeating after every re-boot. So - if that is the case - the interesting question is whether the file vboxdrv.sh could have a simple modification made to it which would allow the 3 signed .ko files to remain untouched on re-boot; or whether there is anything else that could be done - such as placing the 3 files in another location - which would allow them to remain unchanged on re-boot. Any thoughts on this would be much appreciated.

Virtualbox version 6.1.40 r154048 (Qt5.15.3).

[milverton]

There is a workaround which solves my original issue, and that is to have a script run automatically after the machine boots. The script does the first step of signing the kernel module files and the second step of modprobe vboxdrv. I used the cron method of adding the script, but I see that the idea of doing it that way is somewhat controversial. My reference on that is to be found at https://askubuntu.com/questions/814/how ... n-start-up.

So, to recap on this issue:
When the machine is re-booted, the kernel module .ko files apparently get deleted and re-built from scratch, leaving them unsigned. (This is the point I had no idea about when I started this topic.)
With a MOK (machine owner's key) they can be signed. vboxdrv can then be loaded.
To avoid a need to do these steps manually before starting VirtualBox, they can be placed in a script which runs automatically after the machine boots.
There is more than one way of doing that.
The method works with different kernels. (The one I use most of the time right now is 5.19.0-051900-generic, since Ubuntu's update 5.19.0-32-generic kernel revived an old issue of trouble with S3 sleep on my machine.)

When the machine has started up it seems necessary to wait a few seconds before running VirtualBox. With an eye on the clock on the screen, it takes about 12 seconds for that to switch from UTC to local time and the preferred time format I have set up. I would generally wait an equal number of seconds after that before trying to start VirtualBox.

[scottgus1]

I'm no Linux guru. I've seen your posts here and have had no idea what to do to help. But a thought comes to mind: you seem to be the only person suffering from this. Are you doing something unusual. like running self-built Virtualbox or Test Builds or some such? These regular after-host-boot settings aren't normally required.

[milverton]

It's a valid question, scottgus1, but I'm not aware of anything I've done which is out of the ordinary, except eliminating snap, which seems to be something there's a body of support for doing, and threatens the extent of support for Ubuntu. I've never delved into anything in Virtualbox except what I've written about in these posts - and that involves really only one file, vboxdrv.sh, which I experimented with a little bit but ended up restoring to its original state.

My Ubuntu setup is dual-boot with Windows, in its own partition, but since that's something expressly discussed under installing Ubuntu there can't be anything exceptional about that.

Are you saying that you're unable to reproduce the underlying issue, which is that if secure boot is enabled on the Ubuntu host machine, VirtualBox VMs do not start?

Here is a reference on that which many people in other forums seem to often come back to, even recently: https://stegard.net/2016/10/virtualbox- ... untu-fail/

[scottgus1]

I don't have a Linux host to test on, personally. Just checking if something unusual was being done, which apparently isn't happening.

We'll need a Linux guru to weigh in here.

[fth0]

We'll need a Linux guru to weigh in here.

The VirtualBox Linux scripts (e.g. vboxdrv.sh, vboxadd.sh) have been modified several times during the VirtualBox 6.1 release updates. I don't have enough experience in this area, but I'll try and ping someone who could give a hint if he finds time to have a look ...

[milverton]

Thank you both, fth0 and scottgus1: any hint will be much appreciated

[Evar]

I'm a new Rocky Linux 9.1 user and seem to be having the same issue while i have secure boot activated. I installed VirtualBox 7 from the official depot just today and need to sign and modprobe after every single reboot to run VMs in VirtualBox.

The guide i followed is the stegard one mentioned here earlier.

I haven't dug deeper into this yet, but i'll try to spend some time on this tomorrow. If nothing else works, i bet the script could just be set to run on logon in addition to modprobe as a workaround.

[milverton]

For confirmation I'm posting here the version of the file vboxdrv.sh which my VirtualBox installation - version 6.1.40 r154048 (Qt5.15.3) - contains.

[fth0]

FWIW, on my Linux Mint host with Secure Boot enabled, the VirtualBox kernel modules get built and signed during their initial installation and after each Linux kernel update, but never after a reboot.

Additionally, the largest kernel module vboxdrv.ko has a size of ~600 kB (no compression, 6.1.42 + 7.0.6).

[milverton]

That is interesting, fth0. Is the path to the 3 (or 4) .ko files similar to what it would be in Ubuntu, namely /lib/modules/$(uname -r)/misc? (Even allowing for file system differences between Ubuntu and Mint, if they're in a different place - not in a misc directory - that would be interesting in itself.)

Presumably when they were signed a key would have had to be enrolled first with your computer's EFI BIOS, which is a special procedure involving a restart - isn't it??. Do you know what kind of key it was, and whose signature it has? In my case I've found that signing the files adds 439 bytes to the length of each one.

Also, would you mind confirming that the .ko files are actually signed - i.e. do actually have signatures appended to them. While I was struggling to understand this I think I came across a web page which said that the files don't necessarily get a signature added to them, even when there's something like a progress window which says they're being signed. I'll go back through my research and see if I can locate that source, and see if it's relevant in this context.

I can't be quite sure, but I think I had my Ubuntu all installed, complete with VIrtualBox and my VMs, for a considerable length of time before becoming interested in enabling secure boot. To my knowledge the "Ubuntu Live" USB flash drive commonly used to install Ubuntu is inherently dependent on secure boot being disabled for the installation; but as always there could be something I missed.

[Evar]

Booted up the machine this morning and checked the timestamps of the .ko files before going on another reboot.

Code: Select all

# ls -la
total 4996
drwxr-xr-x. 2 root root      66 mars   9 17:11 .
drwxr-xr-x. 8 root root    4096 mars  10 07:44 ..
-rw-r--r--. 1 root root 3791936 mars  10 07:44 vboxdrv.ko
-rw-r--r--. 1 root root  468520 mars  10 07:44 vboxnetadp.ko
-rw-r--r--. 1 root root  844464 mars  10 07:44 vboxnetflt.ko

After reboot i checked and timestamp for the .ko files had changed. I then also ran the signing script and modprobe to see if it still works.

Code: Select all

[root@test ~]# cd /lib/modules
[root@test modules]# ls
5.14.0-162.18.1.el9_1.x86_64  5.14.0-162.6.1.el9_1.x86_64
[root@test modules]# cd 5.14.0-162.18.1.el9_1.x86_64/
[root@test 5.14.0-162.18.1.el9_1.x86_64]# cd misc
[root@test misc]# ls -la
total 4996
drwxr-xr-x. 2 root root      66 mars   9 17:11 .
drwxr-xr-x. 8 root root    4096 mars  10 07:55 ..
-rw-r--r--. 1 root root 3791936 mars  10 07:55 vboxdrv.ko
-rw-r--r--. 1 root root  468520 mars  10 07:55 vboxnetadp.ko
-rw-r--r--. 1 root root  844464 mars  10 07:55 vboxnetflt.ko
[root@test misc]# cd /root/signed-modules/
[root@test signed-modules]# ls
MOK.der  MOK.priv  sign-virtual-box
[root@test signed-modules]# modprobe vboxdrv
modprobe: ERROR: could not insert 'vboxdrv': Key was rejected by service
[root@test signed-modules]# ./sign-virtual-box 
Signing /lib/modules/5.14.0-162.18.1.el9_1.x86_64/misc/vboxdrv.ko
Signing /lib/modules/5.14.0-162.18.1.el9_1.x86_64/misc/vboxnetadp.ko
Signing /lib/modules/5.14.0-162.18.1.el9_1.x86_64/misc/vboxnetflt.ko
[root@test signed-modules]# modprobe vboxdrv

After signing and modprobe:

Code: Select all

[root@test signed-modules]# cd /lib/modules/5.14.0-162.18.1.el9_1.x86_64/misc/
[root@test misc]# ls -la
total 4996
drwxr-xr-x. 2 root root      66 mars  10 07:57 .
drwxr-xr-x. 8 root root    4096 mars  10 07:55 ..
-rw-r--r--. 1 root root 3792376 mars  10 07:57 vboxdrv.ko
-rw-r--r--. 1 root root  468960 mars  10 07:57 vboxnetadp.ko
-rw-r--r--. 1 root root  844904 mars  10 07:57 vboxnetflt.ko

Adding, that before signing when i run modinfo vboxdrv, it will not show as signed and every time i do sign and modprobe, the modinfo vboxdrv shows the signature info.

Very peculiar.

[milverton]

From what Evar has posted it seems it seems there is a propensity for the .ko files to remain in a "misc" directory - which doesn't seem as if it's what VirtualBox ideally wants to happen.

I have checked the public keys in my kernel by running "cat /proc/keys" as root, and there isn't any mention there of one by VirtualBox or Oracle. To me, that's surprising.