Intermittent high load average on host machine

[tommyj27]

I have been seeing some strange behavior that I wanted to run by the forums. I am using Cacti to monitor systems on my network, including two VirtualBox hosts (Debian Lenny, running PUEL 2.2.4). Each host is running three linux guests (5 Lenny and 1 Gentoo). I have been seeing some very strange changes in load-average patterns on the host machines that seem to coincide with guest reboots. Every night, shortly after 23:00, all of the machines get bounced briefly to set up LVM snapshots for backups. For no apparent reason, the load average sometimes spikes on one of the hosts following these cycles. The high load seems to continue all day until the next reboot cycle; then it either subsides or continues on. Only two of the six guests see a high workload, and I think it is a stretch to even call that "high".

I have included a screen capture of the 24 hour load-average on both machines. You can see that yesterday evening the load on the second machine essentially doubled around the time backups ran. I also attached the log file for the most active guest VM from that host. It is a mail server that was moved onto the host machine yesterday (however, this issue has been ongoing for as long as I've been monitoring it). The VM configuration for this guest is also below (the other guests have similar configs with fewer resources).

If anyone has any ideas, I would love to hear them.


vboxuser@dmz2:/var/vm$ VBoxManage showvminfo oldmail
VirtualBox Command Line Management Interface Version 2.2.4
(C) 2005-2009 Sun Microsystems, Inc.
All rights reserved.

Name: oldmail
Guest OS: Other/Unknown
UUID: 4d996097-27bd-4478-920f-8a0f70cdc90d
Config file: /var/vm/oldmail/oldmail/oldmail.xml
Memory size: 384MB
VRAM size: 8MB
Boot menu mode: message and menu
ACPI: on
IOAPIC: off
PAE: off
Time offset: 0 ms
Hardw. virt.ext: off
Nested Paging: off
VT-x VPID: off
State: running (since 2009-08-07T04:31:36.596000000)
Monitor count: 1
3D Acceleration: off
Floppy: empty
SATA: disabled
IDE Controller: PIIX4
Primary master: /var/vm/oldmail/Snapshots/{b586aeb9-e053-4f40-957e-59ab3eb16573}.vmdk (UUID: b586aeb9-e053-4f40-957e-59ab3eb16573)
DVD: empty
NIC 1: MAC: 080027E7107A, Attachment: Bridged Interface 'eth0', Cable connected: on, Trace: off (file: none), Type: Am79C973, Reported speed: 0 Mbps
NIC 2: disabled
NIC 3: disabled
NIC 4: disabled
NIC 5: disabled
NIC 6: disabled
NIC 7: disabled
NIC 8: disabled
UART 1: disabled
UART 2: disabled
Audio: disabled (Driver: Unknown, Controller: Unknown)
Clipboard Mode: Bidirectional
Video mode: 720x400x0
VRDP: enabled (Address 0.0.0.0, Port 33125, MultiConn: off, ReuseSingleConn: off, Authentication type: external)
USB: disabled

USB Device Filters:

<none>

Available remote USB devices:

<none>

Currently Attached USB Devices:

<none>

Shared folders: <none>

VRDP Connection: not active
Clients so far: 0

Guest:

Statistics update: disabled

Snapshots:

Name: checkpoint (UUID: 775f3954-7f87-4158-8d89-6d8da815712e)

Weird Load Average on VBox Host machines.

weird load average.png (76.26 KiB) Viewed 974 times