virtualbox.org

An application using Oracle XE on LINUX under virtualbox runs in 23 minutes on a Dell XPS 8300 and 47 minutes on a HP Envy 750-170se. The virtualbox version is the same on both systems and the vdi file was copied to the HP from the Dell system.

The system configurations are:

Dell

CPU i7-2600
Memory 16GB
Windows 10

HP

CPU i7-6700
Memory 32GB
Windows 10

Has anyone seen this problem?

Thanks,

Bob

Let me try to summarize the description of the problem:

1. I have two different computers.
2. An application runs differently on the two different computers.
3. Has anyone else seen this?

My quick answer would be, yes, everyone.

If you want to have specific answers, you need to be more specific about the problem description. And I mean for you to provide the necessary logs that detail the computers' configuration. Do the following for each of the computers:

1. Start the VM. Not from a saved or suspended state. Clean start.
2. Take the steps required to generate/observe the error.
3. Shut down the VM. Not suspended, not paused. Shut down.
4. Right-click on the VM in the VirtualBox Manager. Select "Show Log..."
5. Save it (just the first log), ZIP it and attach it in your response (see the "Upload attachment" at the bottom of the form).

The application was run on both the Dell (old system) and HP (new system) after booting both systems.

The application uses AWK to process two groups of files that are to be loaded using sqlldr into an Oracle XE database. Then a series of sql DML commands and a PL/SQL process are executed to restructure the data so that it can be upload to a MySQL database on a web site.

Before the test it was verified that both systems have the same version of virtualbox
The .vdi file was copied from the Dell system to the HP system.
The ip address in the HP system was corrected in the /etc/hosts file to reflect the correct ip address.
Both virtualbox virtual machines were shutdown and the log files deleted.
Both virtualbox virtual machines were started and the application was run.
Both virtualbox virtual machines were shutdown.
The log file for each virtual machine was save and filename with prefixed with either Dell or HP reflecting the system.
The two log files were put into the attached zip file.

The results were:

Dell
Total run time: 14m 29.608s
SQL run time: 12m 55.193s

HP
Total run time: 44m 59.663s
SQL run time: 44m 10.574s

Observation made:

The Awk processing was faster on the HP system
When the SQL processing started on the HP system the windows Task Manager (Performance for disk) was at 90%
while the Dell system the disk utilization would go up and down.

I forgot to add:

Is there some configuration parameter, O/S or BOS, than needs to be changed. The Dell system was upgraded from Windows 7 to Windows 10 where as the HP system came with Windows 10 installed.

Thank you for the logs and for distinguishing them by name. It makes it much easier. Here are the differences between the HP and the Dell systems:

00:00:01.721516   NumCPUs           <integer> = 0x0000000000000004 (4)
00:00:02.138646   NumCPUs           <integer> = 0x0000000000000001 (1)
...
00:00:02.037272 CPUM: Physical host cores: 4
00:00:02.595311 CPUM: Physical host cores: 4

And here I think lies your problem. You have 4 physical cores on both systems. You assign all of your HP's CPUs to the guest, while only one CPU on the Dell. Your HP host is suffering from resource shortage with all sorts of side effects. I would assign a maximum of 3 to both systems. You'd want to leave at least one CPU for the host to do its magic.

00:00:01.721730   VRamSize         <integer> = 0x0000000001000000 (16 777 216, 16 MB)
00:00:02.138860   VRamSize         <integer> = 0x0000000000c00000 (12 582 912, 12 MB)

Both of them are really low. I would increase them both to at least 64 MB.

00:00:01.724787 CPUM: Matched host CPU INTEL 0x6/0x5e/0x3 Intel_Core7_Skylake with CPU DB entry 'Intel Core i7-6700K' (INTEL 0x6/0x5e/0x3 Intel_Core7_Skylake)
00:00:02.141034 CPUM: Matched host CPU INTEL 0x6/0x2a/0x7 Intel_Core7_SandyBridge with CPU DB entry 'Intel Core i7-2635QM' (INTEL 0x6/0x2a/0x7 Intel_Core7_SandyBridge)

Other minute differences include the screen size between the two hosts and the fact that VRDE is enabled on the HP host. But I don't think that those count for the discrepancies observed.

Thanks for the input.
I reran the test on both systems with 1 CPU and then with 3 CPUs and there was little difference on the HP system. The Dell system had a very minor speed up with 3 CPUs.

The timings are:

Dell 1 CPU
Total 14m 32.313s
SQL 13m 5.756s

HP 1 CPU
Total 44m 54.794s
SQL 43m 47.123s

Dell 3 CPUs
Total 14m 9.664s
SQL 13m 4.986s

HP 3 CPUs
Total 44m 23.363s
SQL 43m 37.190s

As before each system was rebooted before running the timing for the application. Since the data files for the application had been updated, the .vdi file was copied from the Dell system to HP system so that the same data would be used.

The logs for each run are in the attached zip file.

If I can not get a handle on what is causing the difference, I will be returning the HP system.

Your input has been greatly appreciated.

Bob

I think you need to take more measurements of ALL aspects
of the system during the benchmarks.

From what you describe, the overall process is fairly I/O intensive.
This means changes to the (virtual) CPU configuration may have little impact.

For a proper benchmark, I would start by gathering both Host and Guest
CPU and I/O loads during the run. Is the disk subsystem on the HP the
same as the Dell, especially where the user data being manipulated resides ?
You also need to check system related activity within both the Host and the Guest.
For example, is the HP system suffering from any paging/swapping - either at
the Host level or the Guest level ?

For the elapsed times to that different there must be something quite
significant affecting the run times on the HP.

Remember, when running virtualised BOTH the Host performance and
the Guest performance need to be investigated.

When the SQL processing started on the HP system the windows Task Manager (Performance for disk) was at 90%
while the Dell system the disk utilization would go up and down.

When I first posted, I had missed the above comment in your earlier posting.
Whilst my initial recommendations still hold true, I would concentrate on
the disk subsystem especially, as that appears to be a noticeable difference.

On both systems, is the OS of the guest and the data being manipulated on
the same physical disk or split between different drives ? Can you run any
stand-alone benchmarks like CrystalMark to check the native performance
of the drives involved ?

For both systems, when monitoring the I/O, is the actual I/O load comparable ?
I would expect that the actual number of I/O's issued (reads and writes) would
very similar (the totals will obviously include system related activity, so they
will not be exactly the same).
The percentage figure is just an indication of how busy the disks are servicing
the I/O requests.

@The Raven
All valid points, except that I think that the OP took out of the equation the Guest performance, by using the same guest.

Still it is worrisome that you see a 3x difference between the two hosts. I would run tests outside of VirtualBox, i.e. industry known benchmarks. Yes, a VirtualBox is the case where you want to use it, but it's not something I would troubleshoot a host performance with.

I think that the OP took out of the equation the Guest performance, by using the same guest.

Agreed ... but I've been caught out too many times with things like that
to take it for granted

My hunch is that there is some significant difference in the I/O subsystem performance.
On what appears to be an I/O intensive task, it is the most likely candidate.
My recommendation to run benchmarks on the real hardware (albeit synthetic) would
at least see if the difference was when running on bare-metal or only when VM's are
involved.

The changes in the (virtual) CPU configuration made almost no difference to the
elapsed time, giving further evidence of an I/O related bottleneck.

Hopefully Bob can get some further tests run and we can help him.

Looking at the difference in processing time the major difference was in the SQL processing. Almost all of the SQL processing involves updating a table. With that I started looking at the Oracle processing and found the physical writes for the HP system were 6.04 msec / write while the Dell system was taking 1.45 msec. From that it appears that the HP system has a problem with writing random 8K blocks coming from a virtualbox VM.

Looking on the Internet I found a post about a very slow VM on virtualbox that was fixed by turning off Host I/O Cache. After checking both the Dell and HP virtualbox configurations I found that Host I/O Cache was disabled on both system. At that point I decided to turn on Host I/O Cache for both systems. After rebooting both systems the application was run.

The results for running the application are:

Dell 3 CPUs configured
Total time 4m 42.855s
SQL time 3m 44.030s

HP 3 CPUs configured
Total time 3m 6.502s
SQL time 2m 31.982s

The bottom line is that it looks like the HP does not handle random 8K direct writes to disk as well as the Dell system does. However, when Host I/O Cache is used the HP systems is slightly faster than the Dell system.

Bob

Some other observations:

On the database level the number of physical writes to the data files for the USERS tablespace were about the same
The storage was similar
Dell 240GB SSD drive for C: and a 7200 RPM 2 TB SATA drive for D: 16GB RAM
HP 240GB SSD drive for C: and a 7200 RPM 1 TB SATA drive for D: 32GB RAM

I was very surprised by the initial performance difference and there maybe some parameter BOS or OS that could improve the direct I/O. The HP was purchased to replace the Dell since the XPS 8300 is limited to 16GB. Given how the vendors put systems together, the Dell XPS 8900 which is a match for the HP envy 750-170 it may have similar performance.

Thanks for all of the input!

Bob

I forgot to add that I did use some bench mark tools for I/O. ATTO for disk speed at the Windows level and dd at the LINUX level and the new system was faster for both. However, they tested sequential reads and writes rather than random writes.

Bob,
Interesting findings, thank you for posting the details.

I see the D: volume (user data) on both systems are disk drives - not SSD -
and that they're both 7200 RPM drives but different capacity.
Are they both the same drives series model - for example WD Scorpio Black ?

It's also interesting to note the performance improvement in the Dell system
when Host I/O caching is set on from when it's not used.
I'm fairly new to VirtualBox (although not to Virtualisation and Hypervisors) so cannot
comment on the EXACT mechanism and net effect when Host I/O caching is enabled.
I would assume though that apart from the obvious buffering by the host, it's actually changing
the nature of the physical writes. The most likely difference would be that it employs
some sort of buffer flushing (either explicitly or allows the O/S to manage it) such that
multiple blocks are written in one I/O, as opposed to just writing one at a time in response
to the guests request when Host I/O caching is set off.
Perhaps the difference in the actual physical I/O profile when Host caching is on is the key ?

That sounds very much like the SATA drive used in the HP system has significant performance problems for small random writes (which FYI is the thing that rotating drives are the least efficient at). This is likely due to the drives being from different vendors and/or product lines. Can you look in the device manager and see what it lists for the drives' manufacturer/model? Neither HP nor Dell make their own drives, they just re-brand drives from Hitachi, Seagate, etc. Hopefully device manager can show the drive's real manufacturer. You should also be able to get this information from the sticker on top of the drive, assuming you can access that without voiding warranties.

If you want to run tests to verify this, I recommend trying a free tool called iometer. Install it on the host and start a test that does 8kB random writes to a file on the hard drive using 16 outstanding I/Os. If this really is just a hard drive difference, then you'll see results that should be similar to what you observed in the VMs. Look at the IOPS (input/output operations per second) numbers on both systems, and I'd expect the HP system would have a noticeably lower number than the Dell system. If that's the case, then the easiest solution is just to enable host I/O caching. This will buffer small writes and push them to the disk as a smaller number of larger write operations, which is much more efficient to handle for both the drive and for the host's I/O subsystem. If the Iometer numbers look similar for both systems then this might be something else, like a different storage controller that virtualbox doesn't work with as well.

One other thing to consider is what else might be running on the host system besides the VM. Did you install the OS and software from scratch, or are you using the software image that the machine shipped with? If you're using the default image, then there's a possibility that there's something running in the background that's slowing down your disk accesses (I've had antivirus software do this to me before). This is a less likely culprit, but it's worth keeping in mind.