Semi-solved: Fedora 13 I/O Freeze

[urilabob]

Hi; I have a problem with a fedora 13 guest freezing on I/O. In more detail, the guest boots OK, but freezes after about an hour.

In the scenario I'm clearest about, I boot the machine and open a terminal for a large install I want to do. I start the terminal and system monitor, and everything looks normal, four cpus heavily loaded from parallel installation tasks (the same problem happens if I run the install in synchronous mode), plenty of available memory etc. After an hour or so, though, when I come back I see the following symptoms:
.the install job is frozen (different place every time). If I click on the terminal it will probably progress a few steps, but then freeze again. After a couple more times, the terminal freezes permanently
.the system monitor is also frozen. If I click on it, it continues on as if nothing had happened (but the cpu usage has dropped to close to zero). There's plenty of physical memory still available, and zero swap showing
.The virtualbox network icon is blinking furiously - much more than in its unfrozen state - though I'm not running anything obvious that would cause this
.I can open new terminal windows (i.e. the gui is OK), but they are also frozen - can't type anything into them
.In this state, the shutdown dialogue appears to work OK, but the shutdown also freezes, so I can only stop the machine by a hard stop
.On reboot, there is very little in the logs. At around the time I guess the freeze started, I see sometimes see some ntpd messages in the /var/log/messages (no_sys_peer and spike_detect, not sure if they're abnormal), then /var/log/messages is completely empty until I reboot (even if it's overnight)
.On the host, while the job is running successfully, I see around 350% cpu use, which is about what I expect (4 cpus allocated to the guest). Once it has frozen, the host sees around 4% cpu use from the guest.

I think the only thing I'm doing unusual network-wise is using bridged networking (I need a fixed IP address so my students can login remotely); this is using a separate physical network adapter to the host. The same configuration has worked perfectly well before, and there's nothing in the host logs about network problems, so I doubt it's the network configuration (besides, it's hard to see how a network-based problem would freeze a terminal running in the guest console).

Oh, and I guess the writethrough of swap to a raw disk partition is unusual too; don't think it's implicated here, though, because the system monitor seems to indicate that swap space hasn't been used by the time the crash occurs.

Any thoughts on how to go about further diagnosing this would be greatly appreciated. At the moment, I haven't the faintest idea where to look next. Previously, I saw some messages in the guest logs about fprintd that didn't look good. Since I don't have a fingerprint scanner, I removed fprintd. This didn't seem to fix anything. so I'm stuck, and term starts today....

Best Wishes
Bob

System setup:
Host:
Hardware: dual 4-core Intel X5472, 16GB
Allocated to guest: 4 cpus, 8GB memory
OS: Fedora 13 64 bit (fully updated)
VirtualBox: 3.2.8
VTX entabled
nested paging disabled
PAE disabled
video memory: 64MB, 1 monitor, 3D/2D/remote display disabled
Main install disk: fixed size normal disk, 44GB over ext4 partition, on lvm per normal fedora install
Swap: writethrough to raw partition (16GB)
Network: adapter 1, PCnet-FAST III (Bridged, host eth1 <-> guest eth0)

Guest:
Fedora 13 64 bit (fully updated)


Resolution: it turned out to be a combination of two separate problems, very slow terminal I/O (presumably some kind of problem in the virtualisation of I/O, though I'm not too clear why this affected even console use) combined with random crashes due to smp. So perryg was right (thank you!). I was finally able to get the install to complete by a combination of:
1. setting the number of virtualbox cpus down to 2 (i.e. the number of cpus, not cores)
2. redirecting all output from the install to files

I'm still very puzzled by this buffering issue. We can run multiple vnc sessions on this guest quite OK (this is a shared machine for a whole class) - not fast, but not unacceptably slow. I can't understand how the I/O load of one install in a terminal session - certainly spewing out a fair bit of output, but not unbelievable amounts - could be so much worse than multiple vnc sessions.

Anyway, it's largely solved, so thank you for all your help with this. If you hear any rumours about the smp issues being solved in subsequent versions, it would be great if it could be posted - I'd really like to be able to devote more resources to the teaching machine.

[urilabob]

I found this in /var/log/audit/audit.log immediately before the last reboot:

type=ANOM_ABEND msg=audit(1283316150.201:25): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:xserver_t:s0-s0:c0.c1023 pid=1973 comm="Xorg" sig=11

Not sure whether it's relevant.

[Perryg]

How many processors? Not cores. Try setting it to the true amount of processors and see if it continues to operate.

[fixedwheel]

How many processors? Not cores.

Host:
Hardware: dual 4-core Intel X5472,

as far as i can see this is two Core2 type quad core Xeon processors, no hyperthread: a total of 8 real cores

IMHO giving 4 CPUs to the guest should be ok

[Perryg]

On metal it more than likely would. It has been my experience that trying to use more than the actual amount of processors (not cores) on cpu intensive guests in a virtual environment causes this problem. Try setting it to (2) and see if it performs properly. anyway it is just a suggestion and I am sure you will got a lot more.

[fixedwheel]

I found this in /var/log/audit/audit.log immediately before the last reboot:

type=ANOM_ABEND msg=audit(1283316150.201:25): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:xserver_t:s0-s0:c0.c1023 pid=1973 comm="Xorg" sig=11

this is on the host or the guest?

can you disable selinux on the system where the audit message happens?

[urilabob]

Thanks folks, I really appreciate the helpfulness of people here.

Re cpus vs cores: the same freeze happens even if I set the number of cpus to 1 (the make just proceeds more slowly), so I don't think it is related.

It happens whether I run the make from within the console, or from a an ssh session. In either case, it appears that the session eventually stops displaying the make output, but it's actually still buffered because by clicking repeatedly on the session (console) or hitting carriage return repeatedly in the window (ssh session) I can get it to display more of the output - until eventually it runs out and the session hangs completely. I know it's buffered output rather than the make just waking up and continuing, because at the same time, the system monitor shows zero cpu activity (this make loads the system pretty heavily). But the I/O system is still frozen. It won't accept any more ssh sessions (connection just hangs), and although I can open new windows on the console, I can't type into them. Even weirder, the system monitor also freezes - but I can get it to continue just as if nothing had happened by clicking on the window.

Fixedwheel, the ANOM_ABEND was in the guest - I can't see anything at all that I can relate to this freeze on the host. It (the message) got repeated on the most recent freeze, so I think it might well be related. However I turned off selinux (permissive mode) and ran the make again, and it still froze the same way.

[fixedwheel]

It (the message) got repeated on the most recent freeze, so I think it might well be related. However I turned off selinux (permissive mode) and ran the make again, and it still froze the same way.

i would try w/ selinux disabled completely (can be done from somewhere on gnome desktop menu IIRC, needs reboot)

[urilabob]

FIxedwheel, I'm sorry, no joy - exactly the same failure even with selinux completely disabled.

My thoughts at this stage: since it fails in the same way whether on a terminal in the console or via ssh, it must be something in common between those that is creating a buffering problem. That rules out X-related stuff, for example, or sshd itself. I'm a bit hazy about internals, but I think both are using dbus. Any ideas on how one might get dbus to log what it is doing?

Thanks and Best Wishes
Bob

[v01d]

I think same issue here.

Freezes / becomes unresponsive after a while. It seems to freeze faster if doing something heavy.
The latest freeze was when package manager started upgrade of > 530 apps. System froze, i had to reset,
now fedora won't boot get kernel panic on init (likely interrupted upgrade of some system libs, apps).

On same virtual box i'm running Windows 7 Ultimate with no problems.

[ender2101]

The same problem exactly. It's starting to be really frustrating.
I have no idea what could be going on.
At first I thought it might have to do something with PulseAudio,
as it tends to happen while playing video, but after having disabled
sound system for guest the problem still occured (though it happenned more rarely than before).

[fixedwheel]

i have a fedora13 32bit guest for some time now: never had any freeze IIRC

For curiosity just installed a fedora13 64bit: netinst iso, no LVM (lack of sense in virtual environment IMHO), ext3 (don't need ext4 features), unchecked "office". (4 CPU 4096MB on a dual E5420 16GB Debian host)
Freeze while installing guest additions and same time selecting "System" -> "Administration" -> "SELinux Management"

On next boot i disabled SELinux at first with "Relabel on next reboot" option checked (and disable firewall)

Guest additions installed ok now, and the whole guest is much faster with SELinux disabled ... i'll report if it freezes again

-edit- confirmed, still freezes

[Perryg]

I have never had a problem with Fedora until F13 (don't even think about F14) other then the normal Fedora problems until the last kernel update. Booted or tried to boot and it blew up with kernel panics to the point is corrupted the entire build. Not sure I am going to mess with Fedora any longer, but if I do I will post the results.

[Perryg]

OK I have a brand new fresh install of F13 x86_64 all up-to-date and Guest additions installed. Disabled SeLinux. Tested transfer of 1 GB file and it was exceptionally fast.
fixedwheel, what is you version of VBox? PUEL, OSE, or from svn? If svn when did you update last. Also which GAS are you installing?

[fixedwheel]

what is you version of VBox?

PUEL 3.2.8 on Debian Squeeze AMD64, GA from the same

BTW now after last fedora update run my fedora13 32bit guest hangs too, but i changed from 1 to 4 CPU after update so i don't know yet what is the cause ...