Extremely slow disk performance ubuntu linux guest Mac OS X 11.6.6

[AndrewA]

Several things have happened between the time my ubuntu linux guest disk performance was acceptable, to now, when it is absolutely horrible. Among those things was an upgrade to Mac OS X 11.6.6.

It appears that the slowness is upon the first access to particular filesystem areas. Subsequent accesses are fast. This leads me to believe either block-level or filesystem-level caches are working. My root filesystem is on /dev/mapper. Nothing about that has changed.

As you can see below, reading 100k 1024 blocks initially is at 468kB/s and subsequent read of the same blocks is 855Mb/s.

$ sudo time dd bs=1024 if=/dev/sda of=/dev/null count=100000 skip=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 219.034 s, 468 kB/s
0.17user 0.20system 3:39.03elapsed 0%CPU (0avgtext+0avgdata 2184maxresident)k
199936inputs+0outputs (0major+95minor)pagefaults 0swaps
$ sudo time dd bs=1024 if=/dev/sda of=/dev/null count=100000 skip=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 0.119797 s, 855 MB/s
0.05user 0.06system 0:00.12elapsed 99%CPU (0avgtext+0avgdata 2072maxresident)k
0inputs+0outputs (0major+94minor)pagefaults 0swaps

dd on the host OS, of the underlying img disk file, is fast; so I doubt hardware problems on the host.

I'm not sure where to start, but I'm going to try running some speed tests and/or run some hardware diagnostics on the host.

However, I wanted to see if anyone is experiencing anything similar using VBOX Linux guests on 10.6.6.

AHCI SATA virtual controller on flat-file 40G disk image.
Linux ubuntu 5.13.0-35-generic #40-Ubuntu SMP Mon Mar 7 08:03:10 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

A.

[AndrewA]

On a different guest linux instance on the same Mac OS X host:
Linux AA 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:44:00 UTC 2018 i686 i686 i686 GNU/Linux

$ sudo time dd bs=1024 if=/dev/sda of=/dev/null count=100000 skip=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 4.81274 s, 21.3 MB/s
0.00user 0.14system 0:04.86elapsed 2%CPU (0avgtext+0avgdata 1880maxresident)k
200592inputs+0outputs (1major+74minor)pagefaults 0swaps

$ sudo time dd bs=1024 if=/dev/sda of=/dev/null count=100000 skip=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 0.164495 s, 623 MB/s
0.02user 0.09system 0:00.16elapsed 70%CPU (0avgtext+0avgdata 1984maxresident)k
8inputs+0outputs (0major+77minor)pagefaults 0swaps

... so perhaps this is some issue with the device driver interacions vs VBOX under recent linux kernels??

[AndrewA]

I've also posted this in the Max OS X Hosts forum:

Several things have happened between the time my ubuntu linux guest disk performance was acceptable, to now, when it is absolutely horrible. Among those things was an upgrade to Mac OS X 11.6.6.

It appears that the slowness is upon the first access to particular filesystem areas. Subsequent accesses are fast. This leads me to believe either block-level or filesystem-level caches are working. My root filesystem is on /dev/mapper. Nothing about that has changed.

As you can see below, reading 100k 1024 blocks initially is at 468kB/s and subsequent read of the same blocks is 855Mb/s.

$ sudo time dd bs=1024 if=/dev/sda of=/dev/null count=100000 skip=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 219.034 s, 468 kB/s
0.17user 0.20system 3:39.03elapsed 0%CPU (0avgtext+0avgdata 2184maxresident)k
199936inputs+0outputs (0major+95minor)pagefaults 0swaps
$ sudo time dd bs=1024 if=/dev/sda of=/dev/null count=100000 skip=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 0.119797 s, 855 MB/s
0.05user 0.06system 0:00.12elapsed 99%CPU (0avgtext+0avgdata 2072maxresident)k
0inputs+0outputs (0major+94minor)pagefaults 0swaps

dd on the host OS, of the underlying img disk file, is fast; so I doubt hardware problems on the host.

AHCI SATA virtual controller on flat-file 40G disk image.
Linux ubuntu 5.13.0-35-generic #40-Ubuntu SMP Mon Mar 7 08:03:10 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

On a different guest linux instance on the same Mac OS X host:
Linux AA 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:44:00 UTC 2018 i686 i686 i686 GNU/Linux

$ sudo time dd bs=1024 if=/dev/sda of=/dev/null count=100000 skip=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 4.81274 s, 21.3 MB/s
0.00user 0.14system 0:04.86elapsed 2%CPU (0avgtext+0avgdata 1880maxresident)k
200592inputs+0outputs (1major+74minor)pagefaults 0swaps

$ sudo time dd bs=1024 if=/dev/sda of=/dev/null count=100000 skip=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 0.164495 s, 623 MB/s
0.02user 0.09system 0:00.16elapsed 70%CPU (0avgtext+0avgdata 1984maxresident)k
8inputs+0outputs (0major+77minor)pagefaults 0swaps

A.

[AndrewA]

After upgrading the guest additions package:

$ sudo time dd bs=1024 if=/dev/sda of=/dev/null count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 12.4395 s, 8.2 MB/s
0.05user 0.18system 0:12.44elapsed 1%CPU (0avgtext+0avgdata 2184maxresident)k
198944inputs+0outputs (0major+95minor)pagefaults 0swaps

$ sudo time dd bs=1024 if=/dev/sda of=/dev/null count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 0.118855 s, 862 MB/s
0.04user 0.06system 0:00.11elapsed 99%CPU (0avgtext+0avgdata 2396maxresident)k
0inputs+0outputs (0major+100minor)pagefaults 0swaps

... the initial read performance is up by an order of magnitude. Still very slow compared to the 4x kernel but seems better.
I'm going to try to set this virtual machine to settings closer to the other VM's (e.g., the network adapter is different, etc) and see what happens.
Would a VBox log be helpful?

[AndrewA]

Hmm, at some point while I was running some of these tests, the 4.x kernel ubuntu VM started printing to the console:

May 26 08:17:45 kernel: [ 484.354148] INFO: task systemd-udevd:471 blocked for more than 120 seconds.
May 26 08:17:45 kernel: [ 484.354166] Tainted: G WC E 4.15.0-43-generic #46-Ubuntu
May 26 08:17:45 kernel: [ 484.354183] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

... so there seems to be some level of interaction between the the VMs perhaps via the block device interfaces provided by VirtualBox???

[fth0]

For experimenting, you could add a second virtual hard disk that is not used during boot. Then experiment with the following commands:

Code: Select all   Expand viewCollapse view

sudo hdparm -t --direct /dev/sdb
sudo hdparm -t /dev/sdb
sudo hdparm -T /dev/sdb


A (zipped) VBox.log file wouldn't hurt.

[granada29]

I ran the dd command you specified and get somewhat similar results to what you posted. A bullseye debian guest jumped from 103MB/s to 980MB/s.

However, I also tried this on physical hardware (again debian bullseye) and saw pretty much the same result.

I am not sure that VirtualBox is player here, but that caching is probably occurring in the linux kernel.

You might try experimenting with the Use Host I/O Cache setting for your virtual disk controller. This may have some effect on your results. I found that setting the option made the initial throughput slightly lower.

[AndrewA]

It's not at all clear to me what is being measured here, or what is happening under the hood, given that it appears that each time I run this test, the reported disk read speed increases.
Clearly there is some caching effect happening somewhere (in the host OS?).

Meanwhile, I'm crossing my fingers, but some combination of having updating the guest additions (which I saw rebuilt the initrd images), tweaking the virtualization options (enabled nested tables), etc seems to have at least improved if not eliminated the problem

root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 794 MB in 3.09 seconds = 256.62 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 1632 MB in 3.04 seconds = 537.31 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 2082 MB in 3.00 seconds = 693.89 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 2358 MB in 3.02 seconds = 781.99 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 2556 MB in 3.04 seconds = 841.22 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 2790 MB in 3.00 seconds = 929.40 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 2890 MB in 3.05 seconds = 948.70 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 3122 MB in 3.00 seconds = 1040.51 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 3200 MB in 3.01 seconds = 1062.73 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 3318 MB in 3.00 seconds = 1105.89 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 3376 MB in 3.07 seconds = 1099.82 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 3442 MB in 3.01 seconds = 1142.22 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 3502 MB in 3.08 seconds = 1135.87 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 3660 MB in 3.01 seconds = 1215.27 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 3730 MB in 3.01 seconds = 1238.18 MB/sec
root@ubuntu:/home/user# hdparm -t --direct /dev/sdb

/dev/sdb:
Timing O_DIRECT disk reads: 3798 MB in 3.01 seconds = 1260.39 MB/sec

[fth0]

Disk performance measurement can be a complex topic, because there could be (at least) 3 caches involved: Inside the physical disk drive, in the host OS, in VirtualBox and in the guest OS. Depending on the host/guest OS disk I/O layering, it can get even more complicated than that.

In one of my own setups, the disk read speeds fluctuate around +/-15%, but the 1st measurement isn't different than the others.

[AndrewA]

Well ... the guest on this mac os box, some after upgrading virtual box and ubuntu is now basically unacceptably slow. Some of the things I've done *seem* to have helped, but it's not really clear to me that they did so.

sudo apt-get update takes several minutes to complete. It's simply not normal.

If anyone has ideas I'm all ears.

[fth0]

I'll agree that your initial measurements indicated unacceptable disk I/O throughput.

You didn't provide a (zipped) VBox.log file so far, which would be a basic means to develop ideas.

[AndrewA]

The host machine is fairly old, 2014 iMac 4Ghz Quad Core i7 running Mac OS Big Sur 11.6.6 spinning hard drive hdparm is in the 20Mb/sec range

The same VBox image running on 2017 Macbook Pro 3.1Ghz Quad Core i7 Mac OS Monterey 12.1 with solid state hard drive has hdparm in the 800-1000Mb/sec

Nothing I can find indicates any hardware errors on the iMac.

VBox-log.zip attached.

[fth0]

The VBox.log file is missing the statistics at the end, because you copied it while the VM was still running. Other than that, I don't see any real problems in the VBox.log file.

The host file system is using APFS, which could contribute to the problem if the host disk is either a HDD or a Fusion Drive (*).

The loading of the VirtualBox Guest Additions seems a bit late (around 00:02:54 and 00:04:43), but perhaps you're starting NFS, PostgreSQL and Web services in the guest OS (guessed from your port forwarding).

Perhaps you can do the following: Start the VM and wait ~10 minutes for the guest OS to settle down. Do some disk measurements and shut down the VM from within the guest OS. Then describe what you did and provide the (zipped) VBox.log file.

(*) APFS + Fusion Drive would be my #1 suspect.

[AndrewA]

(sorry if this is a repeat, I thought I posted this already but it doesn't show up in the thread)...

I decided to boot into hardware test (clean) and recovery mode (for disk utility check of root drive). Disk utility ended up checking 17 APFS snapshots, the first of which emitted multiple warnings or errors stating something like "more than 23040 B-Tree nodes" (it was late night, I didn't write it down). The linux vm's hdd backing file is likely a frequent subject of those snapshots. It's about 40G. So I booted back into OS X, turned off Time Machine automatic backups (to presumably destroy those snapshots)... I'm still not getting great performance, but hdparm now reports > 100MB/sec and the linux guest is at least usable again.

On a probably related note, when I booted into recovery mode on the host, and used diskutility to check the root volume, there were 17 snapshots and the 1st snapshot reported warnings errors about "more than 23040 B-Tree nodes" (it was late night and I didn't write down the exact message). Likely the linux guest hdd file is a frequent subject of those snapshots.

Thanks (again) Time Machine.

I don't know whether 100MB/sec is within striking distance of "expected" performance for /dev/sda on which an lvm root partition is mounted, but it's certainly much better than the original 500 kB/sec or subsequent 20MB/sec numbers I measured. I'm now much less suspicious of the linux guest than of the mac os host, though there may still be some unfortunate unexpected consequences of the interactions between VBox's hdd backing file access pattern and APFS's snapshotting algos for large (~40G) files ...

[AndrewA]

hdparm -t --direct /dev/sda

... started out reporting < 1MB/sec and I've now gotten it into the ~100MB/sec range. On my macbook pro with solid state drive and Monterey 12.1 (vs Big Sur 11.6.6) the exact same VM image reports ~1000MB/sec. Dunno if it's the interaction between host OS and guest OD or the hardware (the "slow" machine is a quad code 4Ghz i7 iMac and the laptop is a quad code 3.1Ghz i7 macbook). Go figure.

How did I get those numbers up? I turned off Time Machine automatic backups. I'm pretty sure there was involvement from the 17 snapshots (created by Time Machine) of that APFS volume that contains the linux host hdd backing file.

I had booted into recovery mode and diskutility check of the root volume checked 17 snapshots, the first of which reported warnings about "more than 23040 B-Tree nodes" (it was late night and I didn't note the exact error, woops).

So ... there is definitely an unfortunate interaction between Time Machine, APFS snapshots, large HDD files, and VBox + linux guest's access patterns on those files.

It would be rather unfortunate if I can't use both Time Machine and VBox linux guests on my root volume.