virtualbox.org

I have a Virtual Box 4.0.4 VM guest of Ubuntu 10.04 64 bit running on a Ubuntu 10.04 64 bit host. It is configured for 8 CPUs and 16GB of RAM. The host machine is a 2 CPU, 4 Core Xeon 3GHz with 32 GB of RAM. I am running my own application that takes advantage of all of the CPU cores (similar to make -j8). This same application when run natively will peg the %CPU at around 790 in top. However, it experiences serious performance issues in the VM and the vboxheadless will only consume around 400% on the host and still have about 400% idle. I noticed that there is a lot of CPU cycles spent in the kernel inside the VM as well (my app is in user space). I have tried many things to improve the performance with no luck (playing with VM settings, static vs. dynamic vmdk, move vmdk file to be on local hd instead of over NFS, etc ...). I was able to capture an oprofile report below. The app "myapp" is the one I am running but you can see from the report a lot of time is spent in the kernel mm area.

samples % app name symbol name
811553 29.2978 myapp myapp
408207 14.7366 vmlinux huge_pmd_unshare
333918 12.0547 vmlinux pmtimer_wait
233503 8.4297 libc-2.11.1.so /lib/libc-2.11.1.so
220669 7.9663 vmlinux skip_prefix
179292 6.4726 vmlinux handle_mm_fault
146912 5.3037 ld-2.11.1.so /lib/ld-2.11.1.so
146553 5.2907 vmlinux iommu_full
64251 2.3195 vmlinux do_wp_page
38998 1.4079 vmlinux zlib_inflate
22874 0.8258 gzip /bin/gzip
21466 0.7749 vmlinux select_task_rq_fair
18865 0.6810 ahci /ahci
14725 0.5316 vmlinux clear_zonelist_oom
13253 0.4784 vmlinux __do_proc_dointvec
10972 0.3961 vmlinux gart_free_coherent
8260 0.2982 vmlinux __remove_shared_vm_struct
6993 0.2525 dash /bin/dash
6262 0.2261 vmlinux zlib_inflate_table
6092 0.2199 vmlinux machine_kexec_prepare
5900 0.2130 vmlinux unregister_kmmio_probe
5384 0.1944 vmlinux mempool_resize
4763 0.1719 vmlinux dup_mm
4233 0.1528 libm-2.11.1.so /lib/libm-2.11.1.so
4038 0.1458 e1000 /e1000

The only thing i have found that fixes this problem is using VMware Player 3.0 instead of Virtual Box 4.0.4 to run my Virtual Machine. In VMware Player the CPU on the host is fully consumed and the application works as expected. I would really like to use Virtual Box and am looking for any ideas or config settings. Below is my output from VBoxManage showvminfo:

Name: SCENIC
Guest OS: Ubuntu (64 bit)
UUID: 72c34e54-f428-4800-bef3-1eb0ebaffa77
Config file: /tmp/VirtualBox VMs/SCENIC/SCENIC.vbox
Snapshot folder: /tmp/VirtualBox VMs/SCENIC/Snapshots
Log folder: /tmp/VirtualBox VMs/SCENIC/Logs
Hardware UUID: 72c34e54-f428-4800-bef3-1eb0ebaffa77
Memory size: 16384MB
Page Fusion: off
VRAM size: 32MB
HPET: off
Chipset: piix3
Firmware: BIOS
Number of CPUs: 8
Synthetic Cpu: off
CPUID overrides: None
Boot menu mode: message and menu
Boot Device (1): Floppy
Boot Device (2): DVD
Boot Device (3): HardDisk
Boot Device (4): Not Assigned
ACPI: on
IOAPIC: on
PAE: on
Time offset: 0 ms
RTC: UTC
Hardw. virt.ext: on
Hardw. virt.ext exclusive: on
Nested Paging: on
Large Pages: on
VT-x VPID: on
State: powered off (since 2011-04-08T19:55:15.000000000)
Monitor count: 1
3D Acceleration: off
2D Video Acceleration: off
Teleporter Enabled: off
Teleporter Port: 0
Teleporter Address:
Teleporter Password:
Storage Controller Name (0): IDE Controller
Storage Controller Type (0): PIIX4
Storage Controller Instance Number (0): 0
Storage Controller Max Port Count (0): 2
Storage Controller Port Count (0): 2
Storage Controller Bootable (0): on
Storage Controller Name (1): SATA Controller
Storage Controller Type (1): IntelAhci
Storage Controller Instance Number (1): 0
Storage Controller Max Port Count (1): 30
Storage Controller Port Count (1): 2
Storage Controller Bootable (1): on
IDE Controller (1, 0): Empty
SATA Controller (1, 0): /tmp/VirtualBox VMs/SCENIC/SCENIC-2011-03-24_15-08-27-disk1.vmdk (UUID: 1d629ad6-880c-4e32-b824-b47eeecd65a0)
NIC 1: MAC: 080027136D49, Attachment: NAT, Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0
NIC 1 Settings: MTU: 0, Socket( send: 64, receive: 64), TCP Window( send:64, receive: 64)
NIC 2: MAC: 0800278D824D, Attachment: Host-only Interface 'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0
NIC 3: disabled
NIC 4: disabled
NIC 5: disabled
NIC 6: disabled
NIC 7: disabled
NIC 8: disabled
Pointing Device: USB Tablet
Keyboard Device: PS/2 Keyboard
UART 1: disabled
UART 2: disabled
Audio: enabled (Driver: ALSA, Controller: AC97)
Clipboard Mode: Bidirectional
VRDE: disabled
USB: enabled

USB Device Filters:

<none>

Available remote USB devices:

<none>

Currently Attached USB Devices:

<none>

Shared folders:

Name: 'shared_data', Host path: '/home/data' (machine mapping), writable

VRDE Connection: not active
Clients so far: 0

Guest:

OS type: Ubuntu_64
Additions run level: 0
Configured memory balloon size: 0 MB

2cpu 4core = 8cpu's, the Guest has 8 cpu's... what about the Host? is it suppose to run on thin air?

The host is doing nothing but running the VM for this one application. There is no GUI running (console only) and the host is currently at about 400% idle.

Still the same question, you're complaining about performance yet you allow the Guest all cores and nothing for the Host which essentially 'runs' the Guest.

I understand your concern. However, the machine has plenty of spare cycles and the same config in VMware Player produces desirable results. I have run the VM guest with only 4 CPUs and I still noticed similar results. The problem really is that the VM running on the host is only consuming half the available CPUs even though it has the ability to use them all.

I believe the VM guest is performing a lot of context switching between kernel and user-space and this is what is slowing it down. I do not believe that my application is doing anything out of the ordinary since it runs fine natively and I noticed the same problem on the VM when compiling the kernel in parallel (make -j8).

Then you need to play with the VM settings such as vt-x, pae, ioapic, disk driver(sata/scsi), etc... each one can severely impact performance.

I have tried playing with the VM settings with no results but there is quite a few config combinations possible. I am hoping I can get a little more insight via this forum into what to focus on based on what I found via profiling or if there is a known issue on that version of the kernel.

Are there still vmware tools and drivers installed? can you pinpoint which process is 'slow' and what its doing?

VMWare Player was not installed when Virtual Box was running (I only switch after exhausting all solutions I could think of). I cannot describe what my app is doing as it is proprietary. It is very CPU intensive and consumes alot of RAM and I/O with the hard drive. I saw similar behavior when when compiling the Linux Kernel with many threads (-j) so I would focus on this for analysis.