100% CPU Usage Guest, strace shows clock_gettime(CLOCK_MONOTONIC

[dbaslm]

Had this issue for a long time but no solution found this far, going to ask again.

Got a VM guest running RHEL 6.9:

Code: Select all

Linux vmrhr12c01.localdomain 2.6.32-696.el6.x86_64 #1 SMP Tue Feb 21 00:53:17 EST 2017 x86_64 x86_64 x86_64 GNU/Linux

Running Virtual Box 5.1.28 on a RHEL Host. The Guest VM is constantly running at 100% CPU on the Host.

Code: Select all

top - 10:17:21 up 52 days, 16 min,  6 users,  load average: 1.02, 1.06, 1.28
Tasks: 491 total,   1 running, 490 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.2%us,  5.1%sy,  0.0%ni, 92.4%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32940048k total, 22968556k used,  9971492k free,   441540k buffers
Swap: 32767992k total,   883392k used, 31884600k free, 13890432k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16789 oracle    15   0 6384m  83m  30m S 101.5  0.3  12:48.09 VirtualBox

Doing a trace of PID 16789 shows the following:

Code: Select all

[root@rhrdb060 Logs]# strace -p 16789
Process 16789 attached - interrupt to quit
clock_gettime(CLOCK_MONOTONIC, {4493862, 257709864}) = 0
clock_gettime(CLOCK_MONOTONIC, {4493862, 257748864}) = 0
poll([{fd=25, events=POLLIN}, {fd=27, events=POLLIN}, {fd=31, events=POLLIN}], 3, 43) = 1 ([{fd=31, revents=POLLIN}])
read(31, "\372", 1)                     = 1
clock_gettime(CLOCK_MONOTONIC, {4493862, 292953864}) = 0
poll([{fd=25, events=POLLIN}, {fd=27, events=POLLIN}, {fd=31, events=POLLIN}], 3, 8) = 1 ([{fd=31, revents=POLLIN}])
read(31, "\372", 1)                     = 1
clock_gettime(CLOCK_MONOTONIC, {4493862, 293083864}) = 0
.
.
.etc etc.

This is the contents of the guest grub.conf file.

Code: Select all

0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux 6 (2.6.32-696.el6.x86_64)
	root (hd0,0)
	kernel /vmlinuz-2.6.32-696.el6.x86_64 ro root=/dev/mapper/vg_vmrhbase69-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M rd_LVM_LV=vg_vmrhbase69/lv_swap rd_LVM_LV=vg_vmrhbase69/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet ioapic=off divider=10 nolapic_timer
	initrd /initramfs-2.6.32-696.el6.x86_64.img

Guest log file attached.

Any ideas? Guest VM is basically unusable.

[Perryg]

change divider=10 to divider=100 and reboot.

[dbaslm]

Alas, not made any difference.

Code: Select all

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4436 oracle    15   0 4015m  84m  30m S 100.7  0.3   4:55.45 VirtualBox

Process 4436 attached - interrupt to quit
clock_gettime(CLOCK_MONOTONIC, {4502489, 65891864}) = 0
clock_gettime(CLOCK_MONOTONIC, {4502489, 65949864}) = 0
poll([{fd=25, events=POLLIN}, {fd=27, events=POLLIN}, {fd=31, events=POLLIN}], 3, 35) = 0 (Timeout)
clock_gettime(CLOCK_MONOTONIC, {4502489, 109203864}) = 0
clock_gettime(CLOCK_MONOTONIC, {4502489, 109245864}) = 0
clock_gettime(CLOCK_MONOTONIC, {4502489, 109364864}) = 0
poll([{fd=25, events=POLLIN}, {fd=27, events=POLLIN}, {fd=31, events=POLLIN}], 3, 91) = 0 (Timeout)
clock_gettime(CLOCK_MONOTONIC, {4502489, 209523864}) = 0
clock_gettime(CLOCK_MONOTONIC, {4502489, 209567864}) = 0
clock_gettime(CLOCK_MONOTONIC, {4502489, 209696864}) = 0
poll([{fd=25, events=POLLIN}, {fd=27, events=POLLIN}, {fd=31, events=POLLIN}], 3, 91) = 1 ([{fd=31, revents=POLLIN}])
.
.
.

[Perryg]

Thank you for testing. You should place the divider back to 10 as that is the suggested approach.

Code: Select all

Divirder=10
 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16789 oracle    15   0 6384m  83m  30m S 101.5  0.3  12:48.09 VirtualBox

Divider=100
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4436 oracle    15   0 4015m  84m  30m S 100.7  0.3   4:55.45 VirtualBox

This test makes me wonder exactly what is using the processor and at a constant 100%. If you pause the guest does the cpu rate go down?

From the hosts terminal type VBoxManage showvminfo <VM Name> --details and post here. Replace <VM Name> with the actual name of the VM. Use " " if the name has a space in it.

[Perryg]

You should really fix this as well.

Code: Select all

00:00:00.506024 Console: VM runtime error: fatal=false, errorID=Linux2618TooOld message="The host I/O cache for at least one controller is disabled. There is a known Linux kernel bug which can lead to kernel oopses under heavy load. To our knowledge this bug affects all 2.6.18 kernels.
00:00:00.506039 Either enable the host I/O cache permanently in the VM settings or switch to a newer host kernel.

[dbaslm]

VM host I/O cache is permanently enabled. I guess this is another bug in Virtual Box, reports this setting incorrectly. You actually picked this up in another post I raised and I replied with the same information.

I've tried some other divider values but nothing improves the absolute dreadful performance of the guest VM. Basically as it stands the VM guest is unusable.

[Perryg]

I don't know what else to tell you. That whole series of kernel had serious race conditions and I have not use a 2.* kernel ( host or guest ) for years. Maybe you can post this on bugtracker and see if the DEVs have a suggestion, or try a newer kernel if one is available to you. I build my own when I run into this kind of issue but that is not for everyone.