[Solved] systemd breaks mounting shared folders via rc.local

Discussions about using Linux guests in VirtualBox.
Post Reply
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

[Solved] systemd breaks mounting shared folders via rc.local

Post by arQon »

W7 host, Ubuntu MATE 16.04 guest, 5.1.30 with GAs installed:
(I'm still on 5.1.30 because I need working sound).

>VBoxManage --version
5.1.30r118389

$ /usr/bin/VBoxClient --version
5.1.30r118389

$ uname -a
Linux tabitha 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

The setup is fairly simple: the guest is set to auto-login, and I invariably have a terminal open in one of the VB shared folders, as well as a file manager window, which are restored at that time by the session manager.

I've been using rc.local to mount my shared folders since 2010, and this arrangement has worked 100% of the time until a few months ago. Since then, an ls in the terminal has occasionally shown it as empty on startup (which it isn't), and following an update from 14.04 to 16.04 today that's now always the case.

Where this gets interesting is that if I just "cd ." in that directory and then ls it again, all the files that are actually there show up. So, clearly this is a race of some kind.

The rc.local is obviously +x and being run, the mountpoints are all set up correctly, etc, etc. In the interests of debugging, I split the mount out into a separate bash script:

Code: Select all

#!/bin/bash

# use fmode=666 to fix text files etc on ntfs folders being considered executable

for tries in {1..5}
do
	mount -t vboxsf -o rw,fmode=666 xfer /xfer
	if [[ $? -eq 0 ]] ; then
		echo VBox mount attempt $tries succeeded >> /tmp/rclocal.log
		ls -l /xfer/ >> /tmp/rclocal.log
		break;
	fi
	echo VBox mount attempt $tries failed >> /tmp/rclocal.log
	sleep 1
done
which shows that the mount succeeds on the first try; and the log file also correctly shows the contents of the shared folder.

rc.local should be run LONG before X starts, let alone the DE, so the mountpoint should be usable by the time the terminal starts. of course, systemd may well have broken even that, since it seems to be intent on breaking pretty much everything :(, but AFAICT it got it right:

Code: Select all

(/var/log/syslog)
Mar 18 06:19:43 tabitha systemd[1]: Starting /etc/rc.local Compatibility...
Mar 18 06:19:43 tabitha systemd[1]: Started /etc/rc.local Compatibility.
Mar 18 06:19:48 tabitha systemd[1]: Reached target Multi-User System.
Mar 18 06:19:48 tabitha systemd[1]: Reached target Graphical Interface.
So, I'm pretty much at a loss here. I've been poking at this on and off for months now when it was "only" unreliable, but now it's consistently broken. (On the bright side, that might make it easier to debug).

AFAICT, using VB's AUTO-mount *doesn't* have this problem, though I've only tried that a couple of times so far. The downside with that of course is that the folders aren't where I want them. (Obviously I could potentially symlink them, but I'd rather get this problem sorted out if at all possible).

Any ideas? I'm happy to provide more info if I missed anything, and can try out test builds if one of the devs has a potential fix.

Thanks.
Attachments
vboxlog.zip
(27.07 KiB) Downloaded 13 times
Last edited by arQon on 24. Mar 2018, 09:51, edited 2 times in total.
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: Bizarre race condition when mounting shared folders via rc.local

Post by arQon »

Updated to 5.2.8r121009, and the problem persists. :(

On a brighter note, sound seems fine to me despite all the problems others are having with it. I only tested it for 5 minutes, but it was perfect during that time.
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: Bizarre race condition when mounting shared folders via rc.local

Post by arQon »

Also tried it on a linux host (also 5.2.8) and unsurprisingly got the same problem, also "fixed" by "cd .", so this is the right forum: it's definitely a guest bug.
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: Bizarre race condition when mounting shared folders via rc.local

Post by arQon »

(probably) unrelated, but I stumbled across what looks to be a bug while banging my head against this wall:

Code: Select all

$ file /usr/lib/VBoxGuestAdditions
/usr/lib/VBoxGuestAdditions: broken symbolic link to /opt/VBoxGuestAdditions-5.2.8/lib/VBoxGuestAdditions

/usr/lib/VBoxGuestAdditions 
lrwxrwxrwx 1 root root 52 Mar 20 10:06 /usr/lib/VBoxGuestAdditions -> /opt/VBoxGuestAdditions-5.2.8/lib/VBoxGuestAdditions
If someone else can confirm that it IS just a broken link and the /opt file really doesn't exist, I'll file a bug for it.

--

There's a few other errors kicking around, but they all look irrelevant:

Code: Select all

vboxadd[620]: VirtualBox Guest Additions: Starting.
vboxadd[620]: /opt/VBoxGuestAdditions-5.2.8/init/vboxadd: 256: /opt/VBoxGuestAdditions-5.2.8/init/vboxadd: cannot create : Directory nonexistent
vboxsf: Successfully loaded version 5.2.8 (interface 0x00010004)
line 256 is

Code: Select all

"${INSTALL_DIR}/init/vboxadd-x11" setup 2>> "${LOG}"
but I have no idea what's going on there since both dirs exist (and aren't on weird mounts or anything like that)
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: Bizarre race condition when mounting shared folders via rc.local

Post by arQon »

okay - it's not solved, but I know what the problem is. \o/

and it is, of course, systemd. sigh.

Throwing a "sleep 10" into /etc/rc.local, the desktop comes up in well under 10 seconds, with the shared folder mount (left open in a caja window) empty. Refreshing that view 5s later shows it as still empty, which is new; and doing so again 10s later it's populated properly.

so, time to find out how to fix systemd's latest piece of stupidity. brb. :P
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: systemd-induced race condition when mounting shared folders via rc.local

Post by arQon »

Well, this has worked once so far, which is once more than the previous 30 or 40 reboots... :P

Code: Select all

$ cat /etc/systemd/system/rc-local.service
[Unit]
 Description=/etc/rc.local Compatibility
 ConditionPathExists=/etc/rc.local
 Before=display-manager.service
 After=network.target

[Service]
 Type=oneshot
 ExecStart=/etc/rc.local start
 TimeoutSec=0
 StandardOutput=tty
 RemainAfterExit=yes

[Install]
 WantedBy=multi-user.target


$ sudo systemctl enable rc-local
Would be premature to mark this as Solved, but at least the problem is clear.

The Before and After barriers are almost certainly sub-optimal, since systemd is too much of a headache for me to deal with right now, so they're just pulled from whatever random threads google offered. I'll try and figure out the ideal placement later, but if someone knows it already that would be much appreciated. In particular, the After is probably waiting for eth0 to come up, when all it *really* needs is just for VBoxService to be running, and so on.
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: Bizarre race condition when mounting shared folders via rc.local

Post by arQon »

Being conservative with the barriers seems to have been fortunate, since subsequent attempts to improve them have all reintroduced the original problem. "After=proc-sys-fs-binfmt_misc.automount" seemed sensible, but apparently not.

there's a noticeable pause of about 1s during boot that wasn't there before, but that's a tiny price to pay for having things actually work properly. :)

"systemctl list-units" shows the shares being mounted VERY early in the process - immediately after / - but also shows them being loaded in alphabetical order rather than the order in rc.local, so I suspect that's VB looking at them, seeing they aren't set to automount, and ignoring them.

I'll see if it stays stable over the next week or two.
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: Bizarre race condition when mounting shared folders via rc.local

Post by arQon »

Moving the Before any earlier than that, even just to "graphical.target" (which is the service immediately prior) results in the old failures.

Also, there are "sf_read_super_aux err=-71" errors in syslog, even with vboxsf added to /etc/modules: I changed the first mount to explicitly use "mount.vboxsf" a while ago, and left the other 3 as "mount -t vboxsf", and since there are 3 errors it looks like that form doesn't work. My (very limited) understanding is that an entry in /etc/modprobe.d/ with "alias fs-vboxsf vboxsf" should fix that, but it doesn't seem to make any difference.

I've removed the After entry, but I expect finding the appropriate value for that would allow the -t form to work properly again.
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: systemd breaks mounting shared folders via rc.local

Post by arQon »

Since I wasn't having any luck with the "Before", I thought it made sense to instead look at finding the earliest "After", to maximise the window in which the script could run. After a little poking at logs, I settled on local-fs.target since that seems to be the precursor to the VB automounts (which certainly couldn't happen BEFORE that :P). I also chose a WantedBy that was somewhat-intelligent rather than just being a random copypaste, and updated the Description to something more accurate.

Code: Select all

$ cat /etc/systemd/system/rc-local.service
[Unit]
 Description=repair systemd's breakage of rc.local-mounting for vboxsf
 ConditionPathExists=/etc/rc.local
 Before=display-manager.service
 After=local-fs.target

[Service]
 Type=oneshot
 ExecStart=/etc/rc.local start
 TimeoutSec=0
 StandardOutput=tty

[Install]
 WantedBy=vboxadd.service
If you use rc.local for anything else as well, I'd suggest creating a new "service" with a better name rather than co-opting rc.local, but since I don't I didn't bother.

It wouldn't surprise me if this could be further improved, but it seems robust and is acceptably fast to me (well under 10s), so that'll do for now.

--

Sidenote: as a bonus, while doing all this i discovered that the 16.04 boot now waits for the network to be online before completing. In this VM, that wait represented more than HALF the total bootup time, for no purpose at all; and that was the cause of the slowdown in the boot process that I'd noticed earlier.

Code: Select all

$ sudo systemctl disable NetworkManager-wait-online.service
Post Reply