Status: VirtualBox Wide-Scale Stability Test Matrix v4.1.6 !

This is for discussing general topics about how to use VirtualBox.
Technologov
Volunteer
Posts: 3342
Joined: 10. May 2007, 16:59
Location: Israel

Status: VirtualBox Wide-Scale Stability Test Matrix v4.1.6 !

Post by Technologov »

I decided once in a while to make a serious Wide-Scale stability and OS coverage testing of VirtualBox to discover rough edges.

Host: Windows XP, 32-bit, Intel Core 2 Q6600, ATI Radeon HD 3850, VBox 3.1.0, VT-x=on.

Most serious problems uncovered:
1. Fedora8/X64 FAILS to install Bug #5410
FIXED in 3.2.0
2. openSUSE 10.3/X64 FAILS to install Bug #5457
FIXED in 3.2.0
3. openSUSE Guest Additions seriously broken in 3.1.0.
FIXED in 3.2.0
4. Fedora 7 and 9 have very serious bugs. Not sure what we can do about it.
Fedora bugs; nothing can be done on VBox part. users: Update Fedora.
5. RHEL 3/4 needs more love. Told to Michael.
FIXED in 3.2.0
6. Windows GA: Many executables refuse to run fom Shared Folders Bug #5732
Not fixed.
7. Multiple VMs with Sound: Running Sound from multiple VMs and switching between VMs fails sound on them all Bug #5733
FIXED in 4.0.0
8. OpenGL on Fedora 12/x64 segfaults (after deactivating SELinux).
FIXED in 4.0.0
9. Fedora 10/11/12 cannot install successfully from DVD, unless user manually ejects DVD. Bug #2680
Fedora bug. Maybe VBox can workaround it by changing default boot order to HDD->CD/DVD
10. old SUSE Linux refuse to start from CD-ROM if VT-x enabled Bug #5407
Not fixed.

VMware 7.0.0 Notes
===================
Under column "Shared Folders/Drag-n-Drop" I have actually tested only drag-n-drop.
Overall VMware 7.0 is found to be very stable release with only few bugs.

update: Forums disallows to post any Excel or ODS files. Added ZIP file. Removed non-topic answers.
======================
Updated on 28.9.2010 to show v3.2.8 test results.

I have completed testing Guest Additions of VirtualBox 3.2.8.
v3.2.8 shows much better results compared to v3.1.0.

OS coverage changes:
*Dropped: Fedora 7/8/9/10/11 - Reason: End-of-Life, (Unofficial Reason: no wish from my side to keep testing & supporting them)
*Added: RHEL 6 BETA1, Fedora 13, openSUSE 11.2, openSUSE 11.3

Good News:
The v3.2.8 release is by far the most stable VirtualBox release I have tried.
It fixed the vast majority of v3.1.0 bugs, and the overall stability feel has improved very much.
If you compare the two spread sheets, you see much more Green Color in v3.2.8 as compared to the previous Multi Color release (v3.1.0).

Bad News:

1. 3D Acceleration is a biggest problem for now;
---------------------
a. openSUSE/KDE3: Running "glxgears" on openSUSE guests makes them run at 500 fps, which is a "software rendering" performance. Bug #7531
b. openSUSE/KDE4: I have tried to activate KDE4 kwin desktop compositing, but this failed, resetting to 2D mode. How to debug? Bug #7531, Bug #4582
c. Red Hat/GNOME: Tried to activate Compiz from GNOME menu, but this failed, resetting to 2D mode. How to debug? Bug #7531
Good news: RHEL 6.0 Final Release fixes the problems with 3D -- and Compiz under GNOME now works fine. (unlike BETA)
d. Windows Vista/ XP x64 3D fail -- Bug #7066
FIXED in 4.0.0

Smaller problems:
2. lack of copy-paste on RHEL3. Bug #7524
WONTFIX; Not supported.
3. CPU hotplug fails badly. Bug #7522
Good news: Works fine in RHEL 6.0 x64 Final release. (feature is disabled in RHEL 6 32-bit)
4. Windows GA: Many executables refuse to run fom Shared Folders Bug #5732

Update: results for v4.0.0 arrived
=============================
with comments for 4.0.4 fixes. (28.03.2011).

Host: Windows 7 x64, Intel Core i7 2600K CPU, 16 GB of RAM, Dual NVIDIA GeForce GTX 560 SLI, VT-x=on. "Technologov's Rig 2011"

OS coverage changes:
RHEL 4.7 -> 4.8, 5.3 -> 5.5, 6 BETA1 -> 6.0 Final
Added SLES 10 SP3, SLES 11 SP1
Added Fedora 14, openSUSE 11.4
Added new Linux family: Debian 4/5/6
Feature coverage changes:
Dropped CPU Hot-plug and VirtIO

..and several new bugs:
Guest Additions: Shared Folders Auto-Mount fails on RHEL4 - Bug #8638
Guest Additions [REGRESSION]: Control exec fails on RHEL4 => Fixed in SVN Bug #8639
Guest Additions: Xorg / mouse becomes unresponsive on SLES 11 => Fixed in SVN Bug #8640, Bug #7946
Guest Additions: VM crash with Compiz 3D and Multi-Monitor enabled Bug #8641
Guest Additions: 3D Acceleration fails on Debian 6.0 Bug #8642
Guest Additions: 3D Acceleration doesn't get installed on SLES 11 Bug #8723
Guest Additions: Autorun.sh fails on multiple distros Bug #8725
Guest Additions: Not installed on Debian 3.1 Bug #8778

Update: incomplete results for v4.1.2 arrived (17.Aug.2011).
======================================
It has one critical bug, which leads to skipping whole 4.1.2 release -- in GuestExecute component.
Score: FAILURE

v4.1.6 (5.Nov.2011)

During the test I tested audio/network/2D/3D on both host and guest OSes. Network: NAT/bridged/UDP tunnel.
Also tested GUI, VBoxSVC, VMM, RDP Server, and more...

OS coverage changes: (vs. 4.0.0 results)
Added Fedora 15
Feature coverage changes:
Added 'View Guest IP Address' and 'Automatic GA install via Autorun', plus added prioritization, so low priority OSes will not get tested against low priority features.

..and several bugs:
New Bugs found as a result of this test run:
1.3D / WDDM: Wildlife HD video fails on Win7 guests. (With Aero, Video doesn't shows up, and without Aero, Windows Media Player 12 simply crashes)
(2D Video Acceleration = disabled, but doesn't matters if enabled) -- bug is reproduced as soon as you install WDDM drivers, even if not activated. Workaround: use VLC media player, which optionally renders into Windows GDI, bypassing DirectX.
To reproduce: Install WDDM inside Win7 + run Windows Media Player 12 with "Wildlife HD video", which is included with Windows7 (documents/videos).VBox Bug #9852
2. 3D / GNOME 3: Serious Rendering problem with Multi-monitor enabled. (Fedora 15) VBox Bug #9853
3. 3D / GNOME 3: GNOME3 menu is invisible in seamless mode. VBox Bug #9855
4. VMM: Cannot boot openSUSE 11.4 with software-virt. VBox Bug #9874

Old bugs, that need another look:

a. Intel HD Audio: cannot lower volume or mute from within Vista/Win7 guests. (works with AC'97 codec, VBox Bug #9556)
b. Multi-Monitor: No reason to activate windows of 2nd monitor, if guest OS is not using it. (Multi-monitor guests only, Bug #6737)
c. VBoxSVC had several random crashed (which leads to several VBox GUI crashes), plus all-then-running VMs could not be snapshotted/hibernated.
d. Autorun -- fails on almost all SUSE guests -- needs more love there... Bug #5509
Also opened a bug against KDE4 KDE Bug #286162
e. 3D / Debian6: Euphoria screensaver (from "rss-glx" package) fails with GNOME, but works with KDE4. Bug #8642
f. Windows GA: Many executables refuse to run fom Shared Folders Bug #5732

Overall I am quite happy with v4.1.6 release. It is mostly stable with only moderate amount of "spot" bugs here-and-there.

Also tested on Debian 6.0 host x64. (network+audio+vboxpython+...)

UPDATE: v4.1.6 Stability Matrix is now editable by whole World:
https://docs.google.com/spreadsheet/ccc ... XlNazJUTGc
(original file I keep in Excel format, for reference. It is tested with LibreOffice)

-Technologov
Attachments
VirtualBox-4.1.6-Stability-Matrix.xls.zip
05.Nov.2011. (v1.0a, +2 downloads)
(5.72 KiB) Downloaded 213 times
VirtualBox-4.0.0-Stability-Matrix.xls.zip
(5.17 KiB) Downloaded 161 times
VirtualBox-3.2.8-Stability-Matrix.zip
(4.45 KiB) Downloaded 183 times
VirtualBox-3.1-Stability-Matrix.zip
Includes 3.1.0 host with 3.1.0 and 3.1.2 Guest Additions.
(4.95 KiB) Downloaded 315 times
VMware-7-Stability-Matrix.zip
(4.74 KiB) Downloaded 289 times
rasker
Posts: 32
Joined: 6. Mar 2009, 15:27

Re: Status: VirtualBox Wide-Scale Stability Test Matrix

Post by rasker »

Technologov: Great work, very comprehensive. I am wondering if the table can be recreated in the virtualbox wiki using restructured text? This might be a good place for people to collaborate on updating it as things change. I couldn't find any software to do it automatically (although twiki has a plugin to import excel spreadsheets). I don't really know enough about restructured text (I know it can do tables) but it must be possible, no?

cheers
R
Technologov
Volunteer
Posts: 3342
Joined: 10. May 2007, 16:59
Location: Israel

Re: Status: VirtualBox Wide-Scale Stability Test Matrix

Post by Technologov »

The wiki idea is good one and may work... but only if people that update the table put a version in the field, like v3.1.2.

The biggest problem is that I don't see how can we edit the wiki.
rasker
Posts: 32
Joined: 6. Mar 2009, 15:27

Re: Status: VirtualBox Wide-Scale Stability Test Matrix

Post by rasker »

Yeah you seem to be right! It is a trac wiki as far as I can tell and I have a login on the virtualbox site but I don't think it actually allows me to do anything. Going from the howtos page in the user docs it seems that one has to email the user mailing list to get something added to the trac wiki documentation pages.

Another option I guess is to use one of the free wiki sites to create a wiki specifically about compatibility of the different versions. Maybe wikispaces or perhaps even better wikia which is mediawiki based?
Technologov
Volunteer
Posts: 3342
Joined: 10. May 2007, 16:59
Location: Israel

Re: Status: VirtualBox Wide-Scale Stability Test Matrix

Post by Technologov »

Yes... 3rd party wiki may be needed. If you want .. go for it ! ...or maybe Google has some web base collaborative spreadsheets for us ?
AntiMatter
Volunteer
Posts: 176
Joined: 2. Nov 2008, 06:48
Primary OS: Ubuntu 12.04
VBox Version: PUEL
Guest OSses: All Windows (x32 & x64), Linux
Location: Canada

Re: Status: VirtualBox Wide-Scale Stability Test Matrix

Post by AntiMatter »

Hi Techno,

That was a huge job you have done there. Thanks very much for carrying out this extensive test grid and for sharing the documentation.
_litz
Posts: 42
Joined: 9. Dec 2009, 23:10
Primary OS: Linux other
VBox Version: OSE other
Guest OSses: Centos 4

Re: Status: VirtualBox Wide-Scale Stability Test Matrix

Post by _litz »

Please add to your grid (for a Centos5 host):

1) SMP unusable on RH EL4/Centos 4 guests (os detects 2nd CPU as having 0 'bogomips') due to speed issues -- runs very slow
2) RH EL4/Centos 4 very slow in general even using 1 cpu. CPU/hdd intensive operations like a DB query to a LOCAL database (not even networked) run about 5% normal speed (yes FIVE percent) with guest OS cpu usage maxed @ 100%.
3) SATA is completely nonfunctional for a RH EL4/Centos 4 guest.

(I have had bugs opened for the above for weeks, so far they have been completely ignored)

- _litz
Technologov
Volunteer
Posts: 3342
Joined: 10. May 2007, 16:59
Location: Israel

Re: Status: VirtualBox Wide-Scale Stability Test Matrix

Post by Technologov »

Test Results Updated.

This tests VirtualBox 3.1.2 vs. VMware 7.0.0
angussf
Posts: 3
Joined: 11. May 2009, 21:24
Primary OS: MS Windows XP
VBox Version: OSE other
Guest OSses: Ubuntu, Windows 7 RC

Re: Status: VirtualBox Wide-Scale Stability Test Matrix

Post by angussf »

Great job :mrgreen: . I'd *_really*_ like to see the reverse tests as well -- Linux hosts of various types (Ubuntu, Fedora, CentOS, openSUSE) hosting for various Windows and *nix guests just to see what makes the best platform on which to run VirtualBox :?: .
Technologov
Volunteer
Posts: 3342
Joined: 10. May 2007, 16:59
Location: Israel

Re: Status: VirtualBox Wide-Scale Stability Test Matrix

Post by Technologov »

Installing a lot of different _host OSes_ is too time consuming, because I can't parallelize this job on my PC.

If I had a lab with 4 identical PCs, I could make it happen.

IMO - the best platform to run on, is the one *you* know best. This will save much more time than actual performance differences.
(example: on DOS computer with 486 CPU *I work* much faster than on an i-phone that has more powerful CPU)
Technologov
Volunteer
Posts: 3342
Joined: 10. May 2007, 16:59
Location: Israel

Re: Status: VirtualBox Wide-Scale Stability Test Matrix

Post by Technologov »

abcuser
Volunteer
Posts: 590
Joined: 10. May 2007, 20:03
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Ubuntu, Windows XP

Re: Status: VirtualBox Wide-Scale Stability Test Matrix

Post by abcuser »

I hope some of Oracle employee see this matrix, you should get the job at Oracle in testing lab!!!
Technologov
Volunteer
Posts: 3342
Joined: 10. May 2007, 16:59
Location: Israel

Re: Status: VirtualBox Wide-Scale Stability Test Matrix

Post by Technologov »

crash0veride wrote very interesting tests here:
http://forums.virtualbox.org/viewtopic.php?f=7&t=31770
Technologov
Volunteer
Posts: 3342
Joined: 10. May 2007, 16:59
Location: Israel

Re: Status: VirtualBox Wide-Scale Stability Test Matrix

Post by Technologov »

Updated for v4.0.0 (with comments for 4.0.4 fixes).
Technologov
Volunteer
Posts: 3342
Joined: 10. May 2007, 16:59
Location: Israel

Re: Status: VirtualBox Wide-Scale Stability Test Matrix

Post by Technologov »

Updated for 4.1.2 (incomplete results)
Post Reply