Status: VirtualBox Wide-Scale Stability Test Matrix v4.1.6 !
Posted: 12. Dec 2009, 03:22
I decided once in a while to make a serious Wide-Scale stability and OS coverage testing of VirtualBox to discover rough edges.
Host: Windows XP, 32-bit, Intel Core 2 Q6600, ATI Radeon HD 3850, VBox 3.1.0, VT-x=on.
Most serious problems uncovered:
1. Fedora8/X64 FAILS to install Bug #5410
FIXED in 3.2.0
2. openSUSE 10.3/X64 FAILS to install Bug #5457
FIXED in 3.2.0
3. openSUSE Guest Additions seriously broken in 3.1.0.
FIXED in 3.2.0
4. Fedora 7 and 9 have very serious bugs. Not sure what we can do about it.
Fedora bugs; nothing can be done on VBox part. users: Update Fedora.
5. RHEL 3/4 needs more love. Told to Michael.
FIXED in 3.2.0
6. Windows GA: Many executables refuse to run fom Shared Folders Bug #5732
Not fixed.
7. Multiple VMs with Sound: Running Sound from multiple VMs and switching between VMs fails sound on them all Bug #5733
FIXED in 4.0.0
8. OpenGL on Fedora 12/x64 segfaults (after deactivating SELinux).
FIXED in 4.0.0
9. Fedora 10/11/12 cannot install successfully from DVD, unless user manually ejects DVD. Bug #2680
Fedora bug. Maybe VBox can workaround it by changing default boot order to HDD->CD/DVD
10. old SUSE Linux refuse to start from CD-ROM if VT-x enabled Bug #5407
Not fixed.
VMware 7.0.0 Notes
===================
Under column "Shared Folders/Drag-n-Drop" I have actually tested only drag-n-drop.
Overall VMware 7.0 is found to be very stable release with only few bugs.
update: Forums disallows to post any Excel or ODS files. Added ZIP file. Removed non-topic answers.
======================
Updated on 28.9.2010 to show v3.2.8 test results.
I have completed testing Guest Additions of VirtualBox 3.2.8.
v3.2.8 shows much better results compared to v3.1.0.
OS coverage changes:
*Dropped: Fedora 7/8/9/10/11 - Reason: End-of-Life, (Unofficial Reason: no wish from my side to keep testing & supporting them)
*Added: RHEL 6 BETA1, Fedora 13, openSUSE 11.2, openSUSE 11.3
Good News:
The v3.2.8 release is by far the most stable VirtualBox release I have tried.
It fixed the vast majority of v3.1.0 bugs, and the overall stability feel has improved very much.
If you compare the two spread sheets, you see much more Green Color in v3.2.8 as compared to the previous Multi Color release (v3.1.0).
Bad News:
1. 3D Acceleration is a biggest problem for now;
---------------------
a. openSUSE/KDE3: Running "glxgears" on openSUSE guests makes them run at 500 fps, which is a "software rendering" performance. Bug #7531
b. openSUSE/KDE4: I have tried to activate KDE4 kwin desktop compositing, but this failed, resetting to 2D mode. How to debug? Bug #7531, Bug #4582
c. Red Hat/GNOME: Tried to activate Compiz from GNOME menu, but this failed, resetting to 2D mode. How to debug? Bug #7531
Good news: RHEL 6.0 Final Release fixes the problems with 3D -- and Compiz under GNOME now works fine. (unlike BETA)
d. Windows Vista/ XP x64 3D fail -- Bug #7066
FIXED in 4.0.0
Smaller problems:
2. lack of copy-paste on RHEL3. Bug #7524
WONTFIX; Not supported.
3. CPU hotplug fails badly. Bug #7522
Good news: Works fine in RHEL 6.0 x64 Final release. (feature is disabled in RHEL 6 32-bit)
4. Windows GA: Many executables refuse to run fom Shared Folders Bug #5732
Update: results for v4.0.0 arrived
=============================
with comments for 4.0.4 fixes. (28.03.2011).
Host: Windows 7 x64, Intel Core i7 2600K CPU, 16 GB of RAM, Dual NVIDIA GeForce GTX 560 SLI, VT-x=on. "Technologov's Rig 2011"
OS coverage changes:
RHEL 4.7 -> 4.8, 5.3 -> 5.5, 6 BETA1 -> 6.0 Final
Added SLES 10 SP3, SLES 11 SP1
Added Fedora 14, openSUSE 11.4
Added new Linux family: Debian 4/5/6
Feature coverage changes:
Dropped CPU Hot-plug and VirtIO
..and several new bugs:
Guest Additions: Shared Folders Auto-Mount fails on RHEL4 - Bug #8638
Guest Additions [REGRESSION]: Control exec fails on RHEL4 => Fixed in SVN Bug #8639
Guest Additions: Xorg / mouse becomes unresponsive on SLES 11 => Fixed in SVN Bug #8640, Bug #7946
Guest Additions: VM crash with Compiz 3D and Multi-Monitor enabled Bug #8641
Guest Additions: 3D Acceleration fails on Debian 6.0 Bug #8642
Guest Additions: 3D Acceleration doesn't get installed on SLES 11 Bug #8723
Guest Additions: Autorun.sh fails on multiple distros Bug #8725
Guest Additions: Not installed on Debian 3.1 Bug #8778
Update: incomplete results for v4.1.2 arrived (17.Aug.2011).
======================================
It has one critical bug, which leads to skipping whole 4.1.2 release -- in GuestExecute component.
Score: FAILURE
v4.1.6 (5.Nov.2011)
During the test I tested audio/network/2D/3D on both host and guest OSes. Network: NAT/bridged/UDP tunnel.
Also tested GUI, VBoxSVC, VMM, RDP Server, and more...
OS coverage changes: (vs. 4.0.0 results)
Added Fedora 15
Feature coverage changes:
Added 'View Guest IP Address' and 'Automatic GA install via Autorun', plus added prioritization, so low priority OSes will not get tested against low priority features.
..and several bugs:
New Bugs found as a result of this test run:
1.3D / WDDM: Wildlife HD video fails on Win7 guests. (With Aero, Video doesn't shows up, and without Aero, Windows Media Player 12 simply crashes)
(2D Video Acceleration = disabled, but doesn't matters if enabled) -- bug is reproduced as soon as you install WDDM drivers, even if not activated. Workaround: use VLC media player, which optionally renders into Windows GDI, bypassing DirectX.
To reproduce: Install WDDM inside Win7 + run Windows Media Player 12 with "Wildlife HD video", which is included with Windows7 (documents/videos).VBox Bug #9852
2. 3D / GNOME 3: Serious Rendering problem with Multi-monitor enabled. (Fedora 15) VBox Bug #9853
3. 3D / GNOME 3: GNOME3 menu is invisible in seamless mode. VBox Bug #9855
4. VMM: Cannot boot openSUSE 11.4 with software-virt. VBox Bug #9874
Old bugs, that need another look:
a. Intel HD Audio: cannot lower volume or mute from within Vista/Win7 guests. (works with AC'97 codec, VBox Bug #9556)
b. Multi-Monitor: No reason to activate windows of 2nd monitor, if guest OS is not using it. (Multi-monitor guests only, Bug #6737)
c. VBoxSVC had several random crashed (which leads to several VBox GUI crashes), plus all-then-running VMs could not be snapshotted/hibernated.
d. Autorun -- fails on almost all SUSE guests -- needs more love there... Bug #5509
Also opened a bug against KDE4 KDE Bug #286162
e. 3D / Debian6: Euphoria screensaver (from "rss-glx" package) fails with GNOME, but works with KDE4. Bug #8642
f. Windows GA: Many executables refuse to run fom Shared Folders Bug #5732
Overall I am quite happy with v4.1.6 release. It is mostly stable with only moderate amount of "spot" bugs here-and-there.
Also tested on Debian 6.0 host x64. (network+audio+vboxpython+...)
UPDATE: v4.1.6 Stability Matrix is now editable by whole World:
https://docs.google.com/spreadsheet/ccc ... XlNazJUTGc
(original file I keep in Excel format, for reference. It is tested with LibreOffice)
-Technologov
Host: Windows XP, 32-bit, Intel Core 2 Q6600, ATI Radeon HD 3850, VBox 3.1.0, VT-x=on.
Most serious problems uncovered:
1. Fedora8/X64 FAILS to install Bug #5410
FIXED in 3.2.0
2. openSUSE 10.3/X64 FAILS to install Bug #5457
FIXED in 3.2.0
3. openSUSE Guest Additions seriously broken in 3.1.0.
FIXED in 3.2.0
4. Fedora 7 and 9 have very serious bugs. Not sure what we can do about it.
Fedora bugs; nothing can be done on VBox part. users: Update Fedora.
5. RHEL 3/4 needs more love. Told to Michael.
FIXED in 3.2.0
6. Windows GA: Many executables refuse to run fom Shared Folders Bug #5732
Not fixed.
7. Multiple VMs with Sound: Running Sound from multiple VMs and switching between VMs fails sound on them all Bug #5733
FIXED in 4.0.0
8. OpenGL on Fedora 12/x64 segfaults (after deactivating SELinux).
FIXED in 4.0.0
9. Fedora 10/11/12 cannot install successfully from DVD, unless user manually ejects DVD. Bug #2680
Fedora bug. Maybe VBox can workaround it by changing default boot order to HDD->CD/DVD
10. old SUSE Linux refuse to start from CD-ROM if VT-x enabled Bug #5407
Not fixed.
VMware 7.0.0 Notes
===================
Under column "Shared Folders/Drag-n-Drop" I have actually tested only drag-n-drop.
Overall VMware 7.0 is found to be very stable release with only few bugs.
update: Forums disallows to post any Excel or ODS files. Added ZIP file. Removed non-topic answers.
======================
Updated on 28.9.2010 to show v3.2.8 test results.
I have completed testing Guest Additions of VirtualBox 3.2.8.
v3.2.8 shows much better results compared to v3.1.0.
OS coverage changes:
*Dropped: Fedora 7/8/9/10/11 - Reason: End-of-Life, (Unofficial Reason: no wish from my side to keep testing & supporting them)
*Added: RHEL 6 BETA1, Fedora 13, openSUSE 11.2, openSUSE 11.3
Good News:
The v3.2.8 release is by far the most stable VirtualBox release I have tried.
It fixed the vast majority of v3.1.0 bugs, and the overall stability feel has improved very much.
If you compare the two spread sheets, you see much more Green Color in v3.2.8 as compared to the previous Multi Color release (v3.1.0).
Bad News:
1. 3D Acceleration is a biggest problem for now;
---------------------
a. openSUSE/KDE3: Running "glxgears" on openSUSE guests makes them run at 500 fps, which is a "software rendering" performance. Bug #7531
b. openSUSE/KDE4: I have tried to activate KDE4 kwin desktop compositing, but this failed, resetting to 2D mode. How to debug? Bug #7531, Bug #4582
c. Red Hat/GNOME: Tried to activate Compiz from GNOME menu, but this failed, resetting to 2D mode. How to debug? Bug #7531
Good news: RHEL 6.0 Final Release fixes the problems with 3D -- and Compiz under GNOME now works fine. (unlike BETA)
d. Windows Vista/ XP x64 3D fail -- Bug #7066
FIXED in 4.0.0
Smaller problems:
2. lack of copy-paste on RHEL3. Bug #7524
WONTFIX; Not supported.
3. CPU hotplug fails badly. Bug #7522
Good news: Works fine in RHEL 6.0 x64 Final release. (feature is disabled in RHEL 6 32-bit)
4. Windows GA: Many executables refuse to run fom Shared Folders Bug #5732
Update: results for v4.0.0 arrived
=============================
with comments for 4.0.4 fixes. (28.03.2011).
Host: Windows 7 x64, Intel Core i7 2600K CPU, 16 GB of RAM, Dual NVIDIA GeForce GTX 560 SLI, VT-x=on. "Technologov's Rig 2011"
OS coverage changes:
RHEL 4.7 -> 4.8, 5.3 -> 5.5, 6 BETA1 -> 6.0 Final
Added SLES 10 SP3, SLES 11 SP1
Added Fedora 14, openSUSE 11.4
Added new Linux family: Debian 4/5/6
Feature coverage changes:
Dropped CPU Hot-plug and VirtIO
..and several new bugs:
Guest Additions: Shared Folders Auto-Mount fails on RHEL4 - Bug #8638
Guest Additions [REGRESSION]: Control exec fails on RHEL4 => Fixed in SVN Bug #8639
Guest Additions: Xorg / mouse becomes unresponsive on SLES 11 => Fixed in SVN Bug #8640, Bug #7946
Guest Additions: VM crash with Compiz 3D and Multi-Monitor enabled Bug #8641
Guest Additions: 3D Acceleration fails on Debian 6.0 Bug #8642
Guest Additions: 3D Acceleration doesn't get installed on SLES 11 Bug #8723
Guest Additions: Autorun.sh fails on multiple distros Bug #8725
Guest Additions: Not installed on Debian 3.1 Bug #8778
Update: incomplete results for v4.1.2 arrived (17.Aug.2011).
======================================
It has one critical bug, which leads to skipping whole 4.1.2 release -- in GuestExecute component.
Score: FAILURE
v4.1.6 (5.Nov.2011)
During the test I tested audio/network/2D/3D on both host and guest OSes. Network: NAT/bridged/UDP tunnel.
Also tested GUI, VBoxSVC, VMM, RDP Server, and more...
OS coverage changes: (vs. 4.0.0 results)
Added Fedora 15
Feature coverage changes:
Added 'View Guest IP Address' and 'Automatic GA install via Autorun', plus added prioritization, so low priority OSes will not get tested against low priority features.
..and several bugs:
New Bugs found as a result of this test run:
1.3D / WDDM: Wildlife HD video fails on Win7 guests. (With Aero, Video doesn't shows up, and without Aero, Windows Media Player 12 simply crashes)
(2D Video Acceleration = disabled, but doesn't matters if enabled) -- bug is reproduced as soon as you install WDDM drivers, even if not activated. Workaround: use VLC media player, which optionally renders into Windows GDI, bypassing DirectX.
To reproduce: Install WDDM inside Win7 + run Windows Media Player 12 with "Wildlife HD video", which is included with Windows7 (documents/videos).VBox Bug #9852
2. 3D / GNOME 3: Serious Rendering problem with Multi-monitor enabled. (Fedora 15) VBox Bug #9853
3. 3D / GNOME 3: GNOME3 menu is invisible in seamless mode. VBox Bug #9855
4. VMM: Cannot boot openSUSE 11.4 with software-virt. VBox Bug #9874
Old bugs, that need another look:
a. Intel HD Audio: cannot lower volume or mute from within Vista/Win7 guests. (works with AC'97 codec, VBox Bug #9556)
b. Multi-Monitor: No reason to activate windows of 2nd monitor, if guest OS is not using it. (Multi-monitor guests only, Bug #6737)
c. VBoxSVC had several random crashed (which leads to several VBox GUI crashes), plus all-then-running VMs could not be snapshotted/hibernated.
d. Autorun -- fails on almost all SUSE guests -- needs more love there... Bug #5509
Also opened a bug against KDE4 KDE Bug #286162
e. 3D / Debian6: Euphoria screensaver (from "rss-glx" package) fails with GNOME, but works with KDE4. Bug #8642
f. Windows GA: Many executables refuse to run fom Shared Folders Bug #5732
Overall I am quite happy with v4.1.6 release. It is mostly stable with only moderate amount of "spot" bugs here-and-there.
Also tested on Debian 6.0 host x64. (network+audio+vboxpython+...)
UPDATE: v4.1.6 Stability Matrix is now editable by whole World:
https://docs.google.com/spreadsheet/ccc ... XlNazJUTGc
(original file I keep in Excel format, for reference. It is tested with LibreOffice)
-Technologov