Semi-solved: Fedora 13 I/O Freeze

[Perryg]

Yeah I lost my 32 bit the 2 days before I lost my 64 bit yesterday. Looks like a clean install makes it work so I am pushing towards a F13 kernel problem and regression in F13 somehow. I have not looked into the diffs. of them yet but something seems strange.

[fixedwheel]

my 64bit fedora 13 was a fresh install done yesterday, netinst iso d/l from yesterday too.

now i switched to 1 CPU, IO-APIC off: boot ok, successful login. Got a ABRT "Automatic Bug-Reporting-Tool" alert while opening a tar.gz archive (note that its not frozen, still up and now running byte-unixbench)

Code: Select all

Package:    	kernel
Latest Crash:	Di 14 Sep 2010 21:24:10 
Command:    	not_applicable
Reason:     	WARNING: at kernel/time/clockevents.c:48 clockevent_delta2ns+0x3a/0x79()
Comment:    	None
Bug Reports:	

and the following ABRT window wants to send a backtrace to Red Hat:

Code: Select all

WARNING: at kernel/time/clockevents.c:48 clockevent_delta2ns+0x3a/0x79()
Hardware name: VirtualBox
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.34.6-54.fc13.x86_64 #1
Call Trace:
 [<ffffffff8104d12f>] warn_slowpath_common+0x7c/0x94
 [<ffffffff8104d15b>] warn_slowpath_null+0x14/0x16
 [<ffffffff81071579>] clockevent_delta2ns+0x3a/0x79
 [<ffffffff81bbd122>] setup_boot_APIC_clock+0x22c/0x476
 [<ffffffff81bbce2d>] APIC_init_uniprocessor+0xf8/0x103
 [<ffffffff81bbacad>] native_smp_prepare_cpus+0x1a0/0x375
 [<ffffffff81bae618>] kernel_init+0x88/0x29e
 [<ffffffff8100aa64>] kernel_thread_helper+0x4/0x10
 [<ffffffff81bae590>] ? kernel_init+0x0/0x29e
 [<ffffffff8100aa60>] ? kernel_thread_helper+0x0/0x10

the 32bit f13 now is on 1 CPU too, and executing byte-unixbench where it froze yesterday in the multi-CPU benchmark part (the single-CPU ran ok)


-edit- both fedora 13 guests: byte-unixbench finished ok with 1 CPU

[Perryg]

This may not be an accurate test since this in on my copy from svn. But the settings should work for you. I will see about running from PUEL here in a minute.

Code: Select all

Oracle VM VirtualBox Command Line Management Interface Version 3.2.51_OSE
(C) 2005-2010 Oracle Corporation
All rights reserved.

Name:            Fedora-13
Guest OS:        Fedora (64 bit)
UUID:            f584a5f2-aa92-40b7-970b-30940ae9b90d
Config file:     /media/External/VirtualBox/Machine/Fedora-13/Fedora-13.xml
Hardware UUID:   f584a5f2-aa92-40b7-970b-30940ae9b90d
Memory size:     1024MB
Page Fusion:     off
VRAM size:       12MB
HPET:            off
Number of CPUs:  2
Synthetic Cpu:   off
CPUID overrides: None
Boot menu mode:  message and menu
Boot Device (1): Floppy
Boot Device (2): DVD
Boot Device (3): HardDisk
Boot Device (4): Not Assigned
ACPI:            on
IOAPIC:          on
PAE:             off
Time offset:     0 ms
RTC:             UTC
Hardw. virt.ext: on
Hardw. virt.ext exclusive: on
Nested Paging:   on
Large Pages:     off
VT-x VPID:       on
State:           running (since 2010-09-14T21:19:43.088000000)
Monitor count:   1
3D Acceleration: off
2D Video Acceleration: off
Teleporter Enabled: off
Teleporter Port: 0
Teleporter Address: 
Teleporter Password: 
Storage Controller Name (0):            IDE Controller
Storage Controller Type (0):            PIIX4
Storage Controller Instance Number (0): 0
Storage Controller Max Port Count (0):  2
Storage Controller Port Count (0):      2
Storage Controller Name (1):            SATA Controller
Storage Controller Type (1):            IntelAhci
Storage Controller Instance Number (1): 0
Storage Controller Max Port Count (1):  30
Storage Controller Port Count (1):      1
IDE Controller (1, 0): Empty
SATA Controller (0, 0): /media/External/VirtualBox/HardDisks/Fedora-13.vdi (UUID: 97bc0c9c-8e41-46ec-b350-12e8c902670f)
NIC 1:           MAC: 080027ED0BD9, Attachment: Bridged Interface 'eth0', Cable connected: on, Trace: off (file: none), Type: virtio, Reported speed: 0 Mbps, Boot priority: 0
NIC 2:           disabled
NIC 3:           disabled
NIC 4:           disabled
NIC 5:           disabled
NIC 6:           disabled
NIC 7:           disabled
NIC 8:           disabled
Pointing Device: USB Tablet
Keyboard Device: PS/2 Keyboard
UART 1:          disabled
UART 2:          disabled
Audio:           enabled (Driver: PulseAudio, Controller: AC97)
Clipboard Mode:  Bidirectional
Video mode:      1184x695x32
USB:             enabled

USB Device Filters:

<none>

Available remote USB devices:

<none>

Currently Attached USB Devices:

<none>

Shared folders:  <none>

VRDP Connection:    not active
Clients so far:     0

Guest:

OS type:                             Linux26_64
Additions run level:                 2
Configured memory balloon size:      0 MB

[Perryg]

Here is PUEL. Download deskstop version yesterday.
800+ files on update.

Code: Select all

Oracle VM VirtualBox Command Line Management Interface Version 3.2.8
(C) 2005-2010 Oracle Corporation
All rights reserved.

Name:            Fedora-13
Guest OS:        Fedora (64 bit)
UUID:            3fadd1c4-c60b-40d3-8319-2cee8278225c
Config file:     /home/perry/.VirtualBox/Machines/Fedora-13/Fedora-13.xml
Hardware UUID:   3fadd1c4-c60b-40d3-8319-2cee8278225c
Memory size:     1024MB
Page Fusion:     off
VRAM size:       12MB
HPET:            off
Number of CPUs:  2
Synthetic Cpu:   off
CPUID overrides: None
Boot menu mode:  message and menu
Boot Device (1): Floppy
Boot Device (2): DVD
Boot Device (3): HardDisk
Boot Device (4): Not Assigned
ACPI:            on
IOAPIC:          on
PAE:             off
Time offset:     0 ms
RTC:             UTC
Hardw. virt.ext: on
Hardw. virt.ext exclusive: on
Nested Paging:   on
Large Pages:     off
VT-x VPID:       on
State:           running (since 2010-09-14T21:34:37.671000000)
Monitor count:   1
3D Acceleration: off
2D Video Acceleration: off
Teleporter Enabled: off
Teleporter Port: 0
Teleporter Address: 
Teleporter Password: 
Storage Controller Name (0):            IDE Controller
Storage Controller Type (0):            PIIX4
Storage Controller Instance Number (0): 0
Storage Controller Max Port Count (0):  2
Storage Controller Port Count (0):      2
Storage Controller Name (1):            SATA Controller
Storage Controller Type (1):            IntelAhci
Storage Controller Instance Number (1): 0
Storage Controller Max Port Count (1):  30
Storage Controller Port Count (1):      1
IDE Controller (1, 0): Empty
SATA Controller (0, 0): /media/External/VirtualBox/HardDisks/Fedora-13.vdi (UUID: 97bc0c9c-8e41-46ec-b350-12e8c902670f)
NIC 1:           MAC: 0800277A0E45, Attachment: Bridged Interface 'eth0', Cable connected: on, Trace: off (file: none), Type: virtio, Reported speed: 0 Mbps, Boot priority: 0
NIC 2:           disabled
NIC 3:           disabled
NIC 4:           disabled
NIC 5:           disabled
NIC 6:           disabled
NIC 7:           disabled
NIC 8:           disabled
Pointing Device: USB Tablet
Keyboard Device: PS/2 Keyboard
UART 1:          disabled
UART 2:          disabled
Audio:           enabled (Driver: PulseAudio, Controller: AC97)
Clipboard Mode:  Bidirectional
Video mode:      1224x697x32
VRDP:            disabled
USB:             enabled

USB Device Filters:

<none>

Available remote USB devices:

<none>

Currently Attached USB Devices:

<none>

Shared folders:  <none>

VRDP Connection:    not active
Clients so far:     0

Guest:

OS type:                             Linux26_64
Additions active:                    yes
Configured memory balloon size:      0 MB

[fixedwheel]

now i was on 2 CPU: both running the byte-unixbench.

32bit just finished ok, 64bit was about to be finished (last run of the last multi-CPU test) when i did VBoxManage showvminfo f13-64
the output is nearly the same as yours:

Code: Select all

Oracle VM VirtualBox Command Line Management Interface Version 3.2.8
(C) 2005-2010 Oracle Corporation
All rights reserved.

Name:            f13-64
Guest OS:        Fedora (64 bit)
UUID:            220227ad-390e-4d5c-b91f-ba67716f8287
Config file:     /u01/vbox/Machines/f13-64/f13-64.xml
Hardware UUID:   220227ad-390e-4d5c-b91f-ba67716f8287
Memory size:     4096MB
Page Fusion:     off
VRAM size:       12MB
HPET:            off
Number of CPUs:  2
Synthetic Cpu:   off
CPUID overrides: None
Boot menu mode:  message and menu
Boot Device (1): Floppy
Boot Device (2): DVD
Boot Device (3): HardDisk
Boot Device (4): Not Assigned
ACPI:            on
IOAPIC:          on
PAE:             off
Time offset:     0 ms
RTC:             UTC
Hardw. virt.ext: on
Hardw. virt.ext exclusive: on
Nested Paging:   on
Large Pages:     off
VT-x VPID:       on
State:           running (since 2010-09-14T20:36:03.107000000)
Monitor count:   1
3D Acceleration: off
2D Video Acceleration: off
Teleporter Enabled: off
Teleporter Port: 0
Teleporter Address: 
Teleporter Password: 
Storage Controller Name (0):            IDE-Controller
Storage Controller Type (0):            PIIX4
Storage Controller Instance Number (0): 0
Storage Controller Max Port Count (0):  2
Storage Controller Port Count (0):      2
Storage Controller Name (1):            SATA-Controller
Storage Controller Type (1):            IntelAhci
Storage Controller Instance Number (1): 0
Storage Controller Max Port Count (1):  30
Storage Controller Port Count (1):      1
IDE-Controller (1, 0): Empty
SATA-Controller (0, 0): /u01/vbox/VDI/f13-64.vdi (UUID: bc71c013-7a45-4c6e-9617-7a9566e40f6d)
NIC 1:           MAC: 080027EC9610, Attachment: Bridged Interface 'eth0', Cable connected: on, Trace: off (file: none), Type: virtio, Reported speed: 0 Mbps, Boot priority: 0
NIC 2:           disabled
NIC 3:           disabled
NIC 4:           disabled
NIC 5:           disabled
NIC 6:           disabled
NIC 7:           disabled
NIC 8:           disabled
Pointing Device: PS/2 Mouse
Keyboard Device: PS/2 Keyboard
UART 1:          disabled
UART 2:          disabled
Audio:           enabled (Driver: ALSA, Controller: AC97)
Clipboard Mode:  Bidirectional
Video mode:      1024x768x32
VRDP:            disabled
USB:             enabled

USB Device Filters:

<none>

Available remote USB devices:

<none>

Currently Attached USB Devices:

<none>

Shared folders:  <none>

VRDP Connection:    not active
Clients so far:     0

Guest:

OS type:                             Linux26_64
Additions active:                    yes
Configured memory balloon size:      0 MB

but guess what happened: its frozen still looping full load on 2 cores in the last run of the last multi-CPU test, should have finished short before the 32bit... i'll kill it now

[Perryg]

What happens if you don't run byte-unixbench?

[fixedwheel]

it looked like running just fine with 2 CPU ...

-edit- you mean when executing VBoxManage showvminfo f13-64 while its running? just booted again after killing: still running ...

[Perryg]

What I mean is I don't seem to see this issue and the only difference is you are running byte-unixbench. I don't know but it might not be able to run on F13 since the builds seem fairly behind to me when I checked it.

[fixedwheel]

no executable delivered: it compiles on the target system and uses its runtime - so i don't see any problem using byte-unixbench as a stress test

[fixedwheel]

Anyway, it's largely solved, so thank you for all your help with this. If you hear any rumours about the smp issues being solved in subsequent versions, it would be great if it could be posted - I'd really like to be able to devote more resources to the teaching machine.

JFTR the SMP issue is not VBox, its Fedora 13 IMHO ... i just had a Debian Lenny 64bit and a Squeeze 32bit guest, 4 CPUs assigned to each one, running byte-unixbench at the same time: no problem.

[Perryg]

I have no doubt that the issue manifested itself because of Fedora. The question is why? A new install worked fine for me but the updated kernel killed all (3) of my F13 installs over the course of 2 days. Kernel panic on reboot then the OS was unbootable. I got really upset and deleted them. Yeah I know I more than likely could have repaired them, but this has happened before and to tell you the truth I am getting some what tired of it. OK I have that off my chest now. Funny thing is the new install actually works faster and better than the previous build (at least for me). So I am thinking they changed something in the kernel and did not find all of the depends. Then the explosion!

[fixedwheel]

The question is why?

Fedora is leet bleeding edge ...

(...) then the OS was unbootable. I got really upset and deleted them. Yeah I know I more than likely could have repaired them, but this has happened before and to tell you the truth I am getting some what tired of it. (...)

BTDT, happened to me some time ago with Fedora 12 IIRC

[fixedwheel]

Fedora 12 32bit just completed byte-unixbench with 6 CPUs assigned

[joekrahn]

I am also experiencing hangs with a Fedora13 x86_64 guest. It happens fairly quick when there is a load on all CPUs. Maybe it is related to the I/O APIC emulation (just a guess). Or, perhaps the nmi_watchdog problem still affects FC13. I will try some kernel flags to see if I can find a fix.

[fixedwheel]

try some kernel flags

now i added: divider=10 nohz=off highres=off

f13 64bit is running on 4 CPUs assigned: the single-CPU tests from byte-unixbench are done and the multi-CPU are still running, looks promising (before already died early in the single-CPU tests) -edit- completed ok!

but it has a huge clock drift