kernel: journal commit I/O error

[DoubleHP]

In guest:

Code: Select all

# dd if=/dev/zero of=/tmp/zerooo bs=1M count=700
700+0 records in
700+0 records out
734003200 bytes (734 MB) copied, 28.9182 s, 25.4 MB/s
[...]

Message from syslogd@leon at Mon Jun  5 16:12:57 2017 ...
leon kernel: journal commit I/O error

Then system is read only.

Some people mentions APID issues, or broken hard disj cable; this issue and other disk related issues started when I installed VB, and I really trust my SSD is good. Al tests performed on raw disk, or from host say the disk is good; but I often have disk failure from guests (either via using raw disk, or VMDK)

Code: Select all

# badblocks -svw /tmp/zerooo
Checking for bad blocks in read-write mode
From block 0 to 564356
Testing with pattern 0xaa: done
Reading and comparing: done
Testing with pattern 0x55: done
Reading and comparing: done
Testing with pattern 0xff: done
Reading and comparing: done
Testing with pattern 0x00: done
Reading and comparing: done
Pass completed, 0 bad blocks found.

[socratis]

I'm not even sure what the question is, you got to do a better job describing the problem. But I mainly wanted to say that you didn't even include the Minimum information needed for assistance. And a ZIPPED VBox.log for a session that you have the problem.

[DoubleHP]

Problem is very simple: dd if=/dev/zero of=/tmp/zerooo bs=1M count=700 => / get remounted RO.

Considering all my guests have similar issues with disk accesses, and host claims the disk is perfectly good, the problematic point has to be VB.

Happens with 32b and 64b guests, using raw disk and.or VMDK.

I previously complained about disk issues from guests; but they were random. Now I have a 100% repro.

Guest:

Code: Select all

# uname -a
Linux leon 2.6.21-2-486 #1 Wed Jul 11 03:17:09 UTC 2007 i686 GNU/Linux
# cat /proc/cmdline
root=UUID=29a21915-0d72-42e2-9985-3a97d4abac67 ro panic=30 vga=791

The log is not long; pushed it here: [Mod edit: Removed offsite link, log was already included later on]

After / was remounted RO, I asked VB to reset the guest.

Guest:
VB 5.1.18 r114002
64b
Linux debian
Debian package

Guest: this guest does not have guest additions; this one is 32b. Other guests are 32b, some have GA, some are using SMP, some are not SMP ... all have various issues accessing disk. This guest is using raw disk access; same bugs occur when using VMDKs. Previous thread about it:
viewtopic.php?f=1&t=80795

This guest is the only 32b guest, and was the most stable; so, I really thought it was related to APIC or ACPI; and it never crashed until today, and now I have a 100% repro (never tried to write so large files in this guest in the past).

https://serverfault.com/questions/23603 ... -i-o-error
here some one says the issue is definitely related to APIC

The only alternative in my case ... it could be bad RAM. I installed VB the same week I changed the mother board and RAM. I did several ramtests, but I know these tests can't check everything.

[socratis]

Problem is very simple ...using raw disk and.or VMDK.

I wouldn't call raw disk access "very simple". Au contraire! It comes with a big, fat, red warning. And BTW, what do you mean "and/or VMDK" ? You have to use a VMDK if you need raw access. Or are you saying that if the disk format is VMDK this happens even if you're not using raw access??? Please clarify.

00:00:00.132700 File system of '/root/VirtualBox VMs/Leon-02/Snapshots' (snapshots) is ext4
00:00:00.132731 File system of '/root/VirtualBox VMs/Leon-02/Leon-old-boot.vdi' is ext4
00:00:00.141380 File system of '/root/VirtualBox VMs/sda_sda4.vmdk' is ext4

Any particular reason you're running as 'root'? A.k.a. the worst possible user to run VirtualBox as?

Code: Select all

00:10:50.083326 PIIX3 ATA: execution time for ATA command 0xc8 was 9 seconds
00:11:39.144400 PIIX3 ATA: execution time for ATA command 0xc8 was 10 seconds
02:16:58.233253 PIIX3 ATA: execution time for ATA command 0xc8 was 12 seconds
259:08:12.592056 PIIX3 ATA: execution time for ATA command 0xc8 was 17 seconds
290:19:49.324251 PIIX3 ATA: execution time for ATA command 0xc8 was 8 seconds
410:52:55.254464 PIIX3 ATA: execution time for ATA command 0xc8 was 8 seconds
506:55:07.806710 PIIX3 ATA: execution time for ATA command 0xc8 was 8 seconds
674:55:11.790779 PIIX3 ATA: execution time for ATA command 0xc8 was 11 seconds
674:56:13.144104 PIIX3 ATA: execution time for ATA command 0xc8 was 8 seconds
746:55:40.900289 PIIX3 ATA: execution time for ATA command 0xc8 was 8 seconds
790:28:50.564641 PIIX3 ATA: execution time for ATA command 0xc8 was 8 seconds
818:55:29.064273 PIIX3 ATA: execution time for ATA command 0xc8 was 14 seconds
818:58:15.110335 PIIX3 ATA: execution time for ATA command 0xc8 was 9 seconds
865:30:51.233129 PIIX3 ATA: execution time for ATA command 0xc8 was 8 seconds
866:53:21.773427 PIIX3 ATA: execution time for ATA command 0xc8 was 9 seconds
1010:57:50.270212 PIIX3 ATA: execution time for ATA command 0xc8 was 11 seconds
1011:05:12.801410 PIIX3 ATA: execution time for ATA command 0xc8 was 8 seconds
1174:28:59.245112 PIIX3 ATA: execution time for ATA command 0xc8 was 9 seconds
1486:29:08.408905 PIIX3 ATA: execution time for ATA command 0xc8 was 17 seconds
1490:53:18.273774 PIIX3 ATA: execution time for ATA command 0xc8 was 8 seconds
1510:29:05.349718 PIIX3 ATA: execution time for ATA command 0xc8 was 13 seconds
1514:53:03.715011 PIIX3 ATA: execution time for ATA command 0xc8 was 13 seconds
1514:55:50.390355 PIIX3 ATA: execution time for ATA command 0xc8 was 16 seconds
1634:19:52.994448 PIIX3 ATA: execution time for ATA command 0xc8 was 8 seconds
1650:59:20.363913 PIIX3 ATA: execution time for ATA command 0xc8 was 8 seconds
1658:53:40.067804 PIIX3 ATA: execution time for ATA command 0xc8 was 11 seconds

I did not manipulate the above at all. At the beginning I thought that I was looking at the VBoxSVC log, because that's the one that starts with funny time marks. So, at the 11 min mark, you get a slow ATA warning. And pretty much the same message, until almost 70 days later. If that's the case for (as you said) "100% repro(ducible)", I'm afraid it's going to be some time until you hear back, positive, negative or otherwise...

And, the immediate next message, 31 hours later:

1689:47:21.906054 PIIX3 ATA: Ctl#1: RESET, DevSel=0 AIOIf=0 CmdIf0=0xc8 (30017217 usec ago) CmdIf1=0x00 (-1 usec ago)

Just some observations at the moment. Nothing more. BTW, is the hard drive that the VM is accessing an internal one or not?

[DoubleHP]

Short and partial answer.

Guest 4 is only using VMDK format (entirely simulated disk), and also has disk issues.

Guest 2 tries to access both physical disks.

But, my main disk is a SSD, and this is the one why stores /tmp in the test.

Then I have a second rotating disk; then I use more small virtual disks to handle boot sequences.

Yes I had 70d uptime, and guest two was the most stable machine of all (oldest kernel, 32b, no APIC). But today, I found a way to make it crash 100% by just creating a large file in /tmp; it's not hitting a bad block; crash occurs after creating about 320 or 380MB.

I tried various other ATA chipsets (all alternatives to PIIX3); PIIX3 is the one recommended for stability.