I suspect something's wrong with the 'use host I/O cache' option (#17573)

[pupertest]

---

ModEdit; related ticket: #17573: Regression: "Use Host I/O cache" setting seems to be inverted in 5.2

---

I'm not sure when it started, maybe 1 or to releases back, but I now have a strong suspicion host cache isn't used anymore. I'm using VBox periodically to test operating systems for certain tasks, so it often goes install/update/install software. As I don't care about possible data loss I always enable host I/O cache, and previously it made noticeable speedup for most operations. Recently though everything is painfully slow. Debian install slows to a crawl. Apt dist-upgrade takes many minutes instead of seconds or a couple minutes (host data rate during guest apt upgrade is in hundreds of KB/s with 100% disk load, which never happened before). I'm not even 100% sure it's only caching problem anymore. I've tried moving .vdis to different disks and even to software raid. The only noticeable improvement was when I moved it to an SSD. But I can't use SSD for VMs, I need quick access to a number of different VMs, I simply can't afford that much SSD space just for occasional testing.

VirtualBox 5.2.8. Host is Win10 x64 1709 (4 cores, 16GB), guests are various linuxes and Windows 7-8-10 (mostly 1-2 cores, 1-4GB, only 1 guest is running at a time). Guest additions could be installed or not - I see this pattern of very slow disk access with 100% disk load on the host in both cases. Windows guests seem to also be affected.

I've checked the guest logs and VBox logs and I'm not seeing anything obvious. As it affects all or at least most VMs it's either the change in recent Vbox releases or a Windows 10 update that broke something.

Has anyone noticed anything similar?

[pupertest]

This issue is still bothering me. VirtualBox updated to 5.2.12, Windows 10 to 1803.
I did a little benchmarking today.
Debian 9.3 vm
Installing a dozen package updates with apt.
Source disk (Snapshot 1) is on HDD, differencing disk (Current state) is on another disk (I've tried this before with all other kinds of storage placement, it doesn't really matter).
cache off 5:58
cache on 15:39
cache off, diff disk on ssd 2:18
cache on, diff disk on ssd 3:35
(cache="use host I/O cache")
That is so not right. Basically turning "use host i/o cache" option makes writes slow as hell, it's even quite noticeable writing to SSD. Just a few months ago "use cache" option instead gave a huge performance boost.
I'm very surprised I'm not seeing other reports of the problem. The only one I could find that looks similar: https://superuser.com/questions/1296927 ... -cache-off
(The poster isn't even on a Windows host! But the problem appears to be the same.)
At least I know I'm not alone. Maybe this affects only specific systems, and maybe most people don't turn host cache on. But how can we diagnose the problem then?

[socratis]

maybe most people don't turn host cache on

Because maybe that's not the default?

You know what you could try? Downgrading VirtualBox to a version that was working differently. You don't remember when you started seeing the changes, so you may have to experiment a little bit further. I suggest using the bisection method, so start with 5.2.12 and 5.1.0, then go to 5.2.0 and 5.1.18, and so on...

[The Raven]

Possibly a duplicate of the following ticket
https://www.virtualbox.org/ticket/17573

[pupertest]

Because maybe that's not the default?

My understanding was that most people who use VirtualBox for testing/demo purposes always enable it (like I do), because the performance gain is so significant (see below).
I was hesitant to downgrade my install for testing previous versions with a bunch of vm configs in place, but I do want to resolve the problem for myself and others.
Here's the results.

5.2.0
cache on 16:30

5.1.10 - wouldn't start the vm with a hardening error (incompatibility with the latest Win10 version, I assume)

5.1.38
cache on 2:39
And here's the expected performance.

I've looked through the 5.2.0 changelog and there's the obvious "Storage: first milestone of the I/O stack redesign landed". So I assume in that redesign "use host I/O cache" option was somehow very negatively affected.
And presumably in 5.1 maintenance releases that redesign isn't implemented.

[socratis]

I was hesitant to downgrade my install for testing previous versions with a bunch of vm configs in place

I do it on average between 3 and 5 times a day, no issues (that I don't expect) mostly...

5.2.0: cache on 16:30
5.1.10 - wouldn't start the vm with a hardening error (incompatibility with the latest Win10 version, I assume)
5.1.38: cache on 2:39

I don't even know what these numbers mean, but when I talked about bisection, this is what I mean:

Test with 5.2.0 OK, 5.2.12 not OK. Then test with 5.2.6 (the middle). If OK, then move to 5.2.8 or 5.2.10. See what's working and what's not, by halving the "distance"/versions between what's working vs. what's not.

It's a faster way compared to testing 5.2.0, 5.2.2, 5.2.4, ..., 5.2.12.

[pupertest]

The numbers are a continuation of a benchmarking an OS update operation in a VM.
Basically all 5.2 versions have the problem (latest, as well as several previous, and I confirmed it for 5.2.0).
Latest 5.1 release (5.1.38) does not.
So the logical conclusion is that the mentioned I/O stack redesign in 5.2 branch is the reason, "use host I/O cache" option is broken in there.

[socratis]

OK, excellent. Let's see what the developers have to say on the subject...