virtualbox.org

My host is Windows Server 2016, I am currently running 150 VMs on Oracle VirtualBox, every VMs is powered by Windows Server 2008 R2 and Windows Server 2012 R2, with 1GB of RAM, 1 vCore, and 25GB of Dynamic Storage.

All the VMs are on a Z:Disk - Logical Drive with 6 x 900GB 10K SAS DISK on RAID 5, P440AR Raid Controller ( without cache battery ).
I am getting constantly high disk response time, ranging from 10000ms to 1,000,000ms, yes, 1 million ms.
*Checked Windows Resources Manager, there is no other software that is running with high usage, all of them are VirtualBox Headless
*High disk response time happens when the disk's write and read is low too, low as in less than 5 MB/s
*Z Disk has page file turned off

Specs:
16 Cores 32 Threads E5-2640 V3 X 2
324GB of ECC MEMORY
2 x 900GB RAID 1+0 as C Drive
6 x 900GB RAID 5 as Z Drive ( Where all the VMs is located )

Does anyone have an idea about this? Any help will be appreciated.

1 million (+) ms doesn't sound right. That's over 16 minutes: surely the guest OS or host OS would time out in a small number of milliseconds, not minutes. Are you sure you don't have your units mixed up? 1 million nanoseconds is 1 microsecond, which would be more like it. If you are getting these numbers from the log the I believe the units are nanosecs.

You can't get I/O errors on a virtual drive, which is just an ordinary file on a host drive - hence performance is dictated by the speed of the physical drive.

So, excessive latency usually means a fault on the host drive.

Is the host drive mechanical or SSD?

And make sure the disk cache is enabled in the VM Storage settings.

Hi there, thank you for your reply, I appreciate it a lot. Yes, that's the number I get from Windows Server 2016 Task Manager, which is the host machine, not the virtual machine. All the VMs are in a 900GB SAS Disk. Regarding the disk cache enabled in the VM Storage settings, is it the " Use Host I/O Cache " under VM settings -> Storage? The speed of the hard disk is around 900MB/s on CrystalDiskMark test.

I have embedded a link for a reference of the Task Manager.
https://ibb.co/GdcPQR3

Start the VM from full normal shutdown, not save-state. Run until you see the problem happen, then shut down the VM from within the VM's OS if possible. If not possible, close the Virtualbox window for the VM with the Power Off option set.

Right-click the VM in the main Virtualbox window's VM list, choose Show Log. Save the far left tab's log, zip it, and post the zip file, using the forum's Upload Attachment tab.

I have shut down all of the VMs and enabled HOST I/O Cache on all of them.
" Start the VM from the full normal shutdown, not save-state. Run until you see the problem happen "

For this solution, I don't think I can do it, because usually, nothing will happen after I launch all the VMs, usually after a few days of all the VMs launch, then only the issue will slowly appear.