I made a home virtualbox server using an HP 420 (Xeon 4-core processor, 32 GB ecc ram), Ubuntu 19.10, and zfs local disk on two non-commercial disks (WD Green 2 tb) as a zfs-mirror. My expectation was that using this machine ONLY for Virtualbox would improve disk performance vs. the mixed load that I had on a prior system. I'm using a single .vdi disk file for each VM. I am testing with ONLY one VM active to avoid additional contention (99.9% idle before start).
When I test storage IO, I lose about 50% of the performance (in Mbps) when I don't use host io cache checked in Virtualbox. When I DO have it checked, performance is significantly improved (although, again, using host cache) and close to host performance. I'm wondering if it would be better for cache performance if I used multiple 1 GB files, say, instead of a single 50 GB vdi file for disk storage or would that just create more overhead for the file-system and OS to manage?
A somewhat related question is why there is such a performance loss on IO in the guest unless I have host cache activated? I allocated a lot of additional ram (16 GB when the 64-bit VM is showing maybe 6 GB free on the guest and 10 GB free on the host) so I believe that the VM would use that for disk cache as efficiently as the host does itself?
I know I didn't provide a complete set of data (no bonnie++, etc.) but I'm looking more for general input / suggestions / etc. based on the specs above.
Thank you in advance!