Page 1 of 1

I/O Error with heavy I/O

Posted: 18. Oct 2010, 11:11
by econtreras
Hi

I use a debian guest system with dynamic hd's, and with hosts as MAC OSX.
When i have a lot of I/O (remove a lot of files and doing merges in subversion) and using a lot of memory (many programs opened and many ssh sessions open) I eventually get IO errors in the kernel ring buffer. Anyone knows what might be the problem? See the ringbuffer output below.

Thanks a lot!
Eduardo

Code: Select all

[    0.664295] TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
[    0.672023] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.672883] TCP: Hash tables configured (established 262144 bind 65536)
[    0.674367] TCP reno registered
[    0.693089] NET: Registered protocol family 1
[    0.697417] checking if image is initramfs... it is
[    1.116753] Switched to high resolution mode on CPU 0
[    1.273707] Freeing initrd memory: 6495k freed
[    1.278354] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    1.280118] audit: initializing netlink socket (disabled)
[    1.281476] type=2000 audit(1287392325.281:1): initialized
[    1.298020] Total HugeTLB memory allocated, 0
[    1.300517] VFS: Disk quotas dquot_6.5.1
[    1.301700] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.303190] msgmni has been set to 3017
[    1.305305] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    1.307500] io scheduler noop registered
[    1.308021] io scheduler anticipatory registered
[    1.309265] io scheduler deadline registered
[    1.312042] io scheduler cfq registered (default)
[    1.313256] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    1.314799] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[    1.317201] pci 0000:00:02.0: Boot video device
[    1.320927] Linux agpgart interface v0.103
[    1.326085] Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
[    1.330284] pnp: the driver 'serial' has been registered
[    1.333421] brd: module loaded
[    1.334702] input: Macintosh mouse button emulation as /class/input/input0
[    1.336049] pnp: the driver 'i8042 kbd' has been registered
[    1.336049] i8042 kbd 00:01: driver attached
[    1.336049] pnp: the driver 'i8042 aux' has been registered
[    1.336049] i8042 aux 00:04: driver attached
[    1.336049] PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 1,12
[    1.343245] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.344053] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.358523] mice: PS/2 mouse device common for all mice
[    1.360459] pnp: the driver 'rtc_cmos' has been registered
[    1.363215] input: AT Translated Set 2 keyboard as /class/input/input1
[    1.364768] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    1.366516] rtc0: alarms up to one day
[    1.377906] cpuidle: using governor ladder
[    1.379040] cpuidle: using governor menu
[    1.380079] No iBFT detected.
[    1.381645] TCP cubic registered
[    1.382525] NET: Registered protocol family 17
[    1.384004] registered taskstats version 1
[    1.387560] rtc_cmos rtc_cmos: setting system clock to 2010-10-18 08:58:46 UTC (1287392326)
[    1.390845] Freeing unused kernel memory: 392k freed
[    1.848664] ACPI: ACPI0007:00 is registered as cooling_device0
[    3.481702] Uniform Multi-Platform E-IDE driver
[    3.483266] ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
[    3.529450] Intel(R) PRO/1000 Network Driver - version 7.3.20-k2-NAPI
[    3.530738] Copyright (c) 1999-2006 Intel Corporation.
[    3.532087] ACPI: PCI Interrupt 0000:00:03.0[A] -> GSI 19 (level, low) -> IRQ 19
[    3.533816] PCI: Setting latency timer of device 0000:00:03.0 to 64
[    4.012047] e1000: 0000:00:03.0: e1000_probe: (PCI:33MHz:32-bit) 08:00:27:b9:08:cd
[    4.052041] ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver
[    4.239875] Floppy drive(s): fd0 is 1.44M
[    4.264116] FDC 0 is a S82078B
[    4.352063] e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
[    4.359627] PIIX4: IDE controller (0x8086:0x7111 rev 0x01) at  PCI slot 0000:00:01.1
[    4.362014] PIIX4: not 100% native mode: will probe irqs later
[    4.363846]     ide0: BM-DMA at 0xd000-0xd007
[    4.365825]     ide1: BM-DMA at 0xd008-0xd00f
[    4.367096] Probing IDE interface ide0...
[    4.657126] hda: VBOX HARDDISK, ATA DISK drive
[    5.328372] hda: host max PIO4 wanted PIO255(auto-tune) selected PIO4
[    5.328640] hda: UDMA/33 mode selected
[    5.331012] Probing IDE interface ide1...
[    6.065555] hdc: VBOX CD-ROM, ATAPI CD/DVD-ROM drive
[    6.743723] hdc: host max PIO4 wanted PIO255(auto-tune) selected PIO4
[    6.743898] hdc: UDMA/33 mode selected
[    6.748118] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
[    6.752371] isa bounce pool size: 16 pages
[    6.754210] ide1 at 0x170-0x177,0x376 on irq 15
[    6.767007] ACPI: PCI Interrupt 0000:00:06.0[A] -> GSI 22 (level, low) -> IRQ 22
[    6.769303] PCI: Setting latency timer of device 0000:00:06.0 to 64
[    6.769328] ohci_hcd 0000:00:06.0: OHCI Host Controller
[    6.771225] ohci_hcd 0000:00:06.0: new USB bus registered, assigned bus number 1
[    6.774058] ohci_hcd 0000:00:06.0: irq 22, io mem 0xf0804000
[    6.852275] usb usb1: configuration #1 chosen from 1 choice
[    6.854041] hub 1-0:1.0: USB hub found
[    6.854041] hub 1-0:1.0: 8 ports detected
[    6.958439] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
[    6.961923] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    6.965401] usb usb1: Product: OHCI Host Controller
[    6.966956] usb usb1: Manufacturer: Linux 2.6.26-1-amd64 ohci_hcd
[    6.968247] usb usb1: SerialNumber: 0000:00:06.0
[    6.982354] No dock devices found.
[    7.000049] SCSI subsystem initialized
[    7.018196] libata version 3.00 loaded.
[    7.100753] hda: max request size: 128KiB
[    7.101833] hda: 16777216 sectors (8589 MB) w/256KiB Cache, CHS=16644/16/63
[    7.105085] hda: cache flushes supported
[    7.106713]  hda: hda1 hda2 hda3
[    7.182399] hdc: ATAPI 32X DVD-ROM drive, 128kB Cache
[    7.184565] Uniform CD-ROM driver Revision: 3.20
[    7.962509] kjournald starting.  Commit interval 5 seconds
[    7.963838] EXT3-fs: mounted filesystem with ordered data mode.
[   10.777933] udevd version 125 started
[   14.014958] input: Power Button (FF) as /class/input/input2
[   14.063651] ACPI: Power Button (FF) [PWRF]
[   14.067634] input: Sleep Button (FF) as /class/input/input3
[   14.104037] ACPI: Sleep Button (FF) [SLPF]
[   14.502976] ACPI: AC Adapter [AC] (on-line)
[   14.625024] ACPI: Battery Slot [BAT0] (battery present)
[   14.781142] ACPI: PCI Interrupt 0000:00:04.0[A] -> GSI 20 (level, low) -> IRQ 20
[   14.793575] vboxguest: major 0, IRQ 20, I/O port d020, MMIO at 00000000f0400000 (size 0x400000)
[   14.798973] vboxguest: Successfully loaded version 3.2.8 (interface 0x00010004)
[   14.914445] piix4_smbus 0000:00:07.0: Found 0000:00:07.0 device
[   14.915690] piix4_smbus 0000:00:07.0: SMB base address uninitialized - upgrade BIOS or use force_addr=0xaddr
[   15.053223] input: PC Speaker as /class/input/input4
[   15.257560] input: ImExPS/2 Generic Explorer Mouse as /class/input/input5
[   15.481246] Error: Driver 'pcspkr' is already registered, aborting...
[   15.701218] pnp: the driver 'parport_pc' has been registered
[   15.701359] parport_pc 00:05: reported by Plug and Play ACPI
[   15.703653] parport_pc 00:05: disabling not supported
[   17.825504] Adding 51904k swap on /dev/hda2.  Priority:-1 extents:1 across:51904k
[   19.352100] EXT3 FS on hda3, internal journal
[   20.624862] aic94xx: Adaptec aic94xx SAS/SATA driver version 1.0.3 loaded
[   20.785148] device-mapper: uevent: version 1.0.3
[   20.800041] device-mapper: ioctl: 4.13.0-ioctl (2007-10-18) initialised: dm-devel@redhat.com
[   21.463294] vboxsf: Successfully loaded version 3.2.8 (interface 0x00010004)
[   21.492615] sf_read_super_aux err=-71
[   21.492913] sf_read_super_aux err=-71
[   21.493059] sf_read_super_aux err=-71
[   24.258586] e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[   27.943280] NET: Registered protocol family 10
[   27.944036] lo: Disabled Privacy Extensions
[   38.854747] eth0: no IPv6 routers present
[   58.980479] warning: `ntpd' uses 32-bit capabilities (legacy support in use)
[   62.389176] mtrr: your processor doesn't support write-combining
[   62.389708] mtrr: your processor doesn't support write-combining
[ 5526.662843] hda: dma_timer_expiry: dma status == 0x21
[ 5536.661869] hda: DMA timeout error
[ 5536.661937] hda: dma timeout error: status=0x48 { DriveReady DataRequest }
[ 5536.661942] ide: failed opcode was: unknown
[ 5536.661958] hda: DMA disabled
[ 5563.262144] ide0: reset: master: error (0x00?)
[ 5650.120515] hda: dma_timer_expiry: dma status == 0x21
[ 5660.120749] hda: DMA timeout error
[ 5660.120832] hda: dma timeout error: status=0x48 { DriveReady DataRequest }
[ 5660.121261] ide: failed opcode was: unknown
[ 5660.121285] hda: DMA disabled
[ 5660.216857] ide0: reset: master: error (0x00?)
[ 5660.217002] end_request: I/O error, dev hda, sector 16371024
[ 5660.217023] Buffer I/O error on device hda3, logical block 2028108
[ 5660.217034] lost page write due to I/O error on hda3
[ 5660.217067] end_request: I/O error, dev hda, sector 16371032
[ 5660.217079] Buffer I/O error on device hda3, logical block 2028109
[ 5660.217090] lost page write due to I/O error on hda3
[ 5660.217104] end_request: I/O error, dev hda, sector 16371040
[ 5660.217116] Buffer I/O error on device hda3, logical block 2028110
[ 5660.217127] lost page write due to I/O error on hda3
[ 5660.217140] end_request: I/O error, dev hda, sector 16371048
[ 5660.217152] Buffer I/O error on device hda3, logical block 2028111
[ 5660.217163] lost page write due to I/O error on hda3
[ 5660.217176] end_request: I/O error, dev hda, sector 16371056
[ 5660.217188] Buffer I/O error on device hda3, logical block 2028112
[ 5660.217199] lost page write due to I/O error on hda3
[ 5660.217212] end_request: I/O error, dev hda, sector 16371064
[ 5660.217224] Buffer I/O error on device hda3, logical block 2028113
[ 5660.217235] lost page write due to I/O error on hda3
[ 5660.217249] end_request: I/O error, dev hda, sector 16371072
[ 5660.217261] Buffer I/O error on device hda3, logical block 2028114
[ 5660.217272] lost page write due to I/O error on hda3
[ 5660.217285] end_request: I/O error, dev hda, sector 16371080
[ 5660.217298] Buffer I/O error on device hda3, logical block 2028115
[ 5660.217309] lost page write due to I/O error on hda3
[ 5660.217322] end_request: I/O error, dev hda, sector 16371088
[ 5660.217335] Buffer I/O error on device hda3, logical block 2028116
[ 5660.217346] lost page write due to I/O error on hda3
[ 5660.217359] end_request: I/O error, dev hda, sector 16371096
[ 5660.217371] Buffer I/O error on device hda3, logical block 2028117
[ 5660.217382] lost page write due to I/O error on hda3
[ 5660.217395] end_request: I/O error, dev hda, sector 16371104
[ 5660.217409] end_request: I/O error, dev hda, sector 16371112
[ 5660.217423] end_request: I/O error, dev hda, sector 16371120
[ 5660.217468] end_request: I/O error, dev hda, sector 16399088
[ 5660.217484] end_request: I/O error, dev hda, sector 16399096
[ 5660.217499] end_request: I/O error, dev hda, sector 16399112
[ 5660.217515] end_request: I/O error, dev hda, sector 16399120
[ 5660.217529] end_request: I/O error, dev hda, sector 16399128
[ 5660.217543] end_request: I/O error, dev hda, sector 16399136
[ 5660.217556] end_request: I/O error, dev hda, sector 16399144
[ 5660.217570] end_request: I/O error, dev hda, sector 16399152
[ 5660.217585] end_request: I/O error, dev hda, sector 16399168
[ 5660.217600] end_request: I/O error, dev hda, sector 16399176
[ 5660.217614] end_request: I/O error, dev hda, sector 16399208
[ 5660.217629] end_request: I/O error, dev hda, sector 16399248
[ 5660.217676] end_request: I/O error, dev hda, sector 16399264
[ 5660.217691] end_request: I/O error, dev hda, sector 16399280
[ 5660.217708] end_request: I/O error, dev hda, sector 16399288
[ 5660.217722] end_request: I/O error, dev hda, sector 16399296
[ 5660.217736] end_request: I/O error, dev hda, sector 16399304
[ 5660.217750] end_request: I/O error, dev hda, sector 16399312
[ 5660.217764] end_request: I/O error, dev hda, sector 16399320
[ 5660.217779] end_request: I/O error, dev hda, sector 16399336
[ 5660.217793] end_request: I/O error, dev hda, sector 16399352
[ 5660.217808] end_request: I/O error, dev hda, sector 16399368
[ 5660.217823] end_request: I/O error, dev hda, sector 16399376
[ 5660.217837] end_request: I/O error, dev hda, sector 16399384
[ 5660.217851] end_request: I/O error, dev hda, sector 16399392
[ 5660.217864] end_request: I/O error, dev hda, sector 16399400
[ 5660.217878] end_request: I/O error, dev hda, sector 16399408
[ 5660.217892] end_request: I/O error, dev hda, sector 16399424
[ 5660.217908] end_request: I/O error, dev hda, sector 16399432
[ 5660.217922] end_request: I/O error, dev hda, sector 16399440
[ 5660.217935] end_request: I/O error, dev hda, sector 16399448
[ 5660.217949] end_request: I/O error, dev hda, sector 16399464
[ 5660.217965] end_request: I/O error, dev hda, sector 16399472
[ 5660.217979] end_request: I/O error, dev hda, sector 16399480
[ 5660.217993] end_request: I/O error, dev hda, sector 16399488
[ 5660.218007] end_request: I/O error, dev hda, sector 16399504
[ 5660.218021] end_request: I/O error, dev hda, sector 16399520
[ 5660.218036] end_request: I/O error, dev hda, sector 16399528
[ 5660.218050] end_request: I/O error, dev hda, sector 16399536
[ 5660.218064] end_request: I/O error, dev hda, sector 16399544
[ 5660.218078] end_request: I/O error, dev hda, sector 16399560
[ 5660.218093] end_request: I/O error, dev hda, sector 16399568
[ 5660.218107] end_request: I/O error, dev hda, sector 16399576
[ 5660.218122] end_request: I/O error, dev hda, sector 16399592
[ 5660.218137] end_request: I/O error, dev hda, sector 16399600
[ 5660.218151] end_request: I/O error, dev hda, sector 16399608
[ 5660.218165] end_request: I/O error, dev hda, sector 16399632
[ 5660.218181] end_request: I/O error, dev hda, sector 16399640
[ 5660.218194] end_request: I/O error, dev hda, sector 16399648
[ 5660.218208] end_request: I/O error, dev hda, sector 16399656
[ 5660.218222] end_request: I/O error, dev hda, sector 16399664
[ 5660.218236] end_request: I/O error, dev hda, sector 16399680
[ 5660.218251] end_request: I/O error, dev hda, sector 16399688
[ 5660.218265] end_request: I/O error, dev hda, sector 16399696
[ 5660.218279] end_request: I/O error, dev hda, sector 16399720
[ 5660.218293] end_request: I/O error, dev hda, sector 16399736
[ 5660.218308] end_request: I/O error, dev hda, sector 16399744
[ 5660.218322] end_request: I/O error, dev hda, sector 16399752
[ 5660.218336] end_request: I/O error, dev hda, sector 16399760
[ 5660.218351] end_request: I/O error, dev hda, sector 16399776
[ 5660.218366] end_request: I/O error, dev hda, sector 16399784
[ 5660.218380] end_request: I/O error, dev hda, sector 16399792
[ 5660.218394] end_request: I/O error, dev hda, sector 16399800
[ 5660.218407] end_request: I/O error, dev hda, sector 16399808
[ 5660.218421] end_request: I/O error, dev hda, sector 16399816
[ 5660.218435] end_request: I/O error, dev hda, sector 16399824
[ 5660.218449] end_request: I/O error, dev hda, sector 16399832
[ 5660.218471] end_request: I/O error, dev hda, sector 16399848
[ 5660.218485] end_request: I/O error, dev hda, sector 16399856
[ 5660.218500] end_request: I/O error, dev hda, sector 16399872
[ 5660.218514] end_request: I/O error, dev hda, sector 16399880
[ 5660.218528] end_request: I/O error, dev hda, sector 16399888
[ 5660.218542] end_request: I/O error, dev hda, sector 16399904
[ 5660.218557] end_request: I/O error, dev hda, sector 16399920
[ 5660.218571] end_request: I/O error, dev hda, sector 16399936
[ 5660.218585] end_request: I/O error, dev hda, sector 16399952
[ 5660.218600] end_request: I/O error, dev hda, sector 16399968
[ 5660.218615] end_request: I/O error, dev hda, sector 16399976
[ 5660.218629] end_request: I/O error, dev hda, sector 16399992
[ 5660.218643] end_request: I/O error, dev hda, sector 16400008
[ 5660.218657] end_request: I/O error, dev hda, sector 16400016
[ 5660.218672] end_request: I/O error, dev hda, sector 16400032
[ 5660.218686] end_request: I/O error, dev hda, sector 16400040
[ 5660.218701] end_request: I/O error, dev hda, sector 16400056
[ 5660.218715] end_request: I/O error, dev hda, sector 16400064
[ 5660.218729] end_request: I/O error, dev hda, sector 16400088
[ 5660.218744] end_request: I/O error, dev hda, sector 16400096
[ 5660.218757] end_request: I/O error, dev hda, sector 16400104
[ 5660.218772] end_request: I/O error, dev hda, sector 16400120
[ 5660.218787] end_request: I/O error, dev hda, sector 16400128
[ 5660.218808] end_request: I/O error, dev hda, sector 16400136
[ 5660.218825] end_request: I/O error, dev hda, sector 16400152
[ 5660.218840] end_request: I/O error, dev hda, sector 16400168
[ 5660.218884] end_request: I/O error, dev hda, sector 16400176
[ 5660.218899] end_request: I/O error, dev hda, sector 16400200
[ 5660.219255] end_request: I/O error, dev hda, sector 16400208
[ 5660.219279] end_request: I/O error, dev hda, sector 16400232
[ 5660.219295] end_request: I/O error, dev hda, sector 16400240
[ 5660.219309] end_request: I/O error, dev hda, sector 16400256
[ 5660.219324] end_request: I/O error, dev hda, sector 16400272
[ 5660.219338] end_request: I/O error, dev hda, sector 16400280
[ 5660.219352] end_request: I/O error, dev hda, sector 16400296
[ 5660.219366] end_request: I/O error, dev hda, sector 16400304
[ 5660.219380] end_request: I/O error, dev hda, sector 16400320
[ 5660.219396] end_request: I/O error, dev hda, sector 16400328
[ 5660.219409] end_request: I/O error, dev hda, sector 16400336
[ 5660.219423] end_request: I/O error, dev hda, sector 16400344
[ 5660.219437] end_request: I/O error, dev hda, sector 16400352
[ 5660.219467] end_request: I/O error, dev hda, sector 16400376
[ 5660.219482] end_request: I/O error, dev hda, sector 16400400
[ 5660.219497] end_request: I/O error, dev hda, sector 16400416
[ 5660.219511] end_request: I/O error, dev hda, sector 16400424
[ 5660.219525] end_request: I/O error, dev hda, sector 16400432
[ 5660.219539] end_request: I/O error, dev hda, sector 16400440
[ 5660.219553] end_request: I/O error, dev hda, sector 16400456
[ 5660.219568] end_request: I/O error, dev hda, sector 16400464
[ 5660.219582] end_request: I/O error, dev hda, sector 16400472
[ 5660.219595] end_request: I/O error, dev hda, sector 16400480
[ 5660.219609] end_request: I/O error, dev hda, sector 16400488
[ 5660.219622] end_request: I/O error, dev hda, sector 16400496
[ 5660.219636] end_request: I/O error, dev hda, sector 16400528
[ 5660.219651] end_request: I/O error, dev hda, sector 16400544
[ 5660.219666] end_request: I/O error, dev hda, sector 16400552
[ 5660.219680] end_request: I/O error, dev hda, sector 16400568
[ 5660.219694] end_request: I/O error, dev hda, sector 16400576
[ 5660.219709] end_request: I/O error, dev hda, sector 16400600
[ 5660.219723] end_request: I/O error, dev hda, sector 16400608
[ 5660.219737] end_request: I/O error, dev hda, sector 16400616
[ 5660.219751] end_request: I/O error, dev hda, sector 16400624
[ 5660.219765] end_request: I/O error, dev hda, sector 16400632
[ 5660.219779] end_request: I/O error, dev hda, sector 16400648
[ 5660.219794] end_request: I/O error, dev hda, sector 16400664
[ 5660.219809] end_request: I/O error, dev hda, sector 16400672
[ 5660.219822] end_request: I/O error, dev hda, sector 16400680
[ 5660.219836] end_request: I/O error, dev hda, sector 16400688
[ 5660.219895] end_request: I/O error, dev hda, sector 16400696
[ 5660.219911] end_request: I/O error, dev hda, sector 16400704
[ 5660.219925] end_request: I/O error, dev hda, sector 16400720
[ 5660.219947] end_request: I/O error, dev hda, sector 16400744
[ 5660.219962] end_request: I/O error, dev hda, sector 16400752
[ 5660.219983] end_request: I/O error, dev hda, sector 16400784
[ 5660.219998] end_request: I/O error, dev hda, sector 16400792
[ 5660.220012] end_request: I/O error, dev hda, sector 16400808
[ 5660.220028] end_request: I/O error, dev hda, sector 16400824
[ 5660.220043] end_request: I/O error, dev hda, sector 16400888
[ 5660.220058] end_request: I/O error, dev hda, sector 16400920
[ 5660.220073] end_request: I/O error, dev hda, sector 16400968
[ 5660.220088] end_request: I/O error, dev hda, sector 16400992
[ 5660.220104] end_request: I/O error, dev hda, sector 16401008
[ 5660.220118] end_request: I/O error, dev hda, sector 16401016
[ 5660.220133] end_request: I/O error, dev hda, sector 16401040
[ 5660.220148] end_request: I/O error, dev hda, sector 16401560
[ 5660.220163] end_request: I/O error, dev hda, sector 16401568
[ 5660.220178] end_request: I/O error, dev hda, sector 16401584
[ 5660.220193] end_request: I/O error, dev hda, sector 16401600
[ 5660.220208] end_request: I/O error, dev hda, sector 16401608
[ 5660.220222] end_request: I/O error, dev hda, sector 16401616
[ 5660.220236] end_request: I/O error, dev hda, sector 16401624
[ 5660.220250] end_request: I/O error, dev hda, sector 16401640
[ 5660.220266] end_request: I/O error, dev hda, sector 16401648
[ 5660.220280] end_request: I/O error, dev hda, sector 16401656
[ 5660.220294] end_request: I/O error, dev hda, sector 16401664
[ 5660.220309] end_request: I/O error, dev hda, sector 16401672
[ 5660.220323] end_request: I/O error, dev hda, sector 16401680
[ 5660.220336] end_request: I/O error, dev hda, sector 16401688
[ 5660.220347] end_request: I/O error, dev hda, sector 16401704
[ 5660.220347] end_request: I/O error, dev hda, sector 16401712
[ 5660.220347] end_request: I/O error, dev hda, sector 16401720
[ 5660.220347] end_request: I/O error, dev hda, sector 16401744
[ 5660.220347] end_request: I/O error, dev hda, sector 16401752
[ 5660.220347] end_request: I/O error, dev hda, sector 16401760
[ 5660.220347] end_request: I/O error, dev hda, sector 16401776
[ 5660.220347] end_request: I/O error, dev hda, sector 16401784
[ 5660.220347] end_request: I/O error, dev hda, sector 16401792
[ 5660.220347] end_request: I/O error, dev hda, sector 16401808
[ 5660.220347] end_request: I/O error, dev hda, sector 16401816
[ 5660.220347] end_request: I/O error, dev hda, sector 16401832
[ 5660.220347] end_request: I/O error, dev hda, sector 16401848
[ 5660.220347] end_request: I/O error, dev hda, sector 16401864
[ 5660.220347] end_request: I/O error, dev hda, sector 16401872
[ 5660.220347] end_request: I/O error, dev hda, sector 16401888
[ 5660.220347] end_request: I/O error, dev hda, sector 16401920
[ 5660.220347] end_request: I/O error, dev hda, sector 16401928
[ 5660.220347] end_request: I/O error, dev hda, sector 16401960
[ 5660.220347] end_request: I/O error, dev hda, sector 16402752
[ 5660.220347] end_request: I/O error, dev hda, sector 16402760
[ 5660.220347] end_request: I/O error, dev hda, sector 16579400
[ 5660.220347] end_request: I/O error, dev hda, sector 16579496
[ 5660.220347] end_request: I/O error, dev hda, sector 16579704
[ 5660.220347] end_request: I/O error, dev hda, sector 16579832
[ 5660.220347] end_request: I/O error, dev hda, sector 16579928
[ 5660.220347] end_request: I/O error, dev hda, sector 16580176
[ 5660.220347] end_request: I/O error, dev hda, sector 16580392
[ 5660.220347] end_request: I/O error, dev hda, sector 16580568
[ 5660.220347] end_request: I/O error, dev hda, sector 16581232
[ 5660.220347] end_request: I/O error, dev hda, sector 16582016
[ 5660.220347] end_request: I/O error, dev hda, sector 16582392
[ 5660.220347] end_request: I/O error, dev hda, sector 16582664
[ 5660.220347] end_request: I/O error, dev hda, sector 16582800
[ 5660.220347] end_request: I/O error, dev hda, sector 16582984
[ 5660.220347] end_request: I/O error, dev hda, sector 16583184
[ 5660.220347] end_request: I/O error, dev hda, sector 16583360
[ 5660.220347] end_request: I/O error, dev hda, sector 16583496
[ 5660.220347] end_request: I/O error, dev hda, sector 16583800
[ 5660.220347] end_request: I/O error, dev hda, sector 16584008
[ 5660.220347] end_request: I/O error, dev hda, sector 16584256
[ 5660.220347] end_request: I/O error, dev hda, sector 16584352
[ 5660.220347] end_request: I/O error, dev hda, sector 16584536
[ 5660.220347] end_request: I/O error, dev hda, sector 16584760
[ 5660.220347] end_request: I/O error, dev hda, sector 16584976
[ 5660.220347] end_request: I/O error, dev hda, sector 16585112
[ 5660.220347] end_request: I/O error, dev hda, sector 16585424
[ 5660.220347] end_request: I/O error, dev hda, sector 16585520
[ 5660.220347] end_request: I/O error, dev hda, sector 16585616
[ 5660.220347] end_request: I/O error, dev hda, sector 16585712
[ 5660.226725] end_request: I/O error, dev hda, sector 16585880
[ 5660.226759] end_request: I/O error, dev hda, sector 16586048
[ 5660.226768] end_request: I/O error, dev hda, sector 16587584
[ 5660.226777] end_request: I/O error, dev hda, sector 16589208
[ 5660.226796] end_request: I/O error, dev hda, sector 16589304
[ 5660.226806] end_request: I/O error, dev hda, sector 16589744
[ 5660.226815] end_request: I/O error, dev hda, sector 16590040
[ 5660.226824] end_request: I/O error, dev hda, sector 16590296
[ 5660.226841] end_request: I/O error, dev hda, sector 16590552
[ 5660.226851] end_request: I/O error, dev hda, sector 16590688
[ 5660.226860] end_request: I/O error, dev hda, sector 16590912
[ 5660.226869] end_request: I/O error, dev hda, sector 16591688
[ 5660.226891] end_request: I/O error, dev hda, sector 16592136
[ 5660.226901] end_request: I/O error, dev hda, sector 16592504
[ 5660.226911] end_request: I/O error, dev hda, sector 16592688
[ 5660.226920] end_request: I/O error, dev hda, sector 16592848
[ 5660.226937] end_request: I/O error, dev hda, sector 16593168
[ 5660.226947] end_request: I/O error, dev hda, sector 16593240
[ 5660.226956] end_request: I/O error, dev hda, sector 16593312
[ 5660.226965] end_request: I/O error, dev hda, sector 16593384
[ 5660.226982] end_request: I/O error, dev hda, sector 16593456
[ 5660.226992] end_request: I/O error, dev hda, sector 16593528
[ 5660.227001] end_request: I/O error, dev hda, sector 16593600
[ 5660.227010] end_request: I/O error, dev hda, sector 16593672
[ 5660.227027] end_request: I/O error, dev hda, sector 16593744
[ 5660.227037] end_request: I/O error, dev hda, sector 16593872
[ 5660.227046] end_request: I/O error, dev hda, sector 16593944
[ 5660.227055] end_request: I/O error, dev hda, sector 16594016
[ 5660.227077] end_request: I/O error, dev hda, sector 16594088
[ 5660.227087] end_request: I/O error, dev hda, sector 16594160
[ 5660.227096] end_request: I/O error, dev hda, sector 16594232
[ 5660.227105] end_request: I/O error, dev hda, sector 16594304
[ 5660.227122] end_request: I/O error, dev hda, sector 16594560
[ 5660.227132] end_request: I/O error, dev hda, sector 16594784
[ 5660.227141] end_request: I/O error, dev hda, sector 16595032
[ 5660.227150] end_request: I/O error, dev hda, sector 16595264
[ 5660.227167] end_request: I/O error, dev hda, sector 16595384
[ 5660.227177] end_request: I/O error, dev hda, sector 16595568
[ 5660.227186] end_request: I/O error, dev hda, sector 16595904
[ 5660.227195] end_request: I/O error, dev hda, sector 16596112
[ 5660.227279] end_request: I/O error, dev hda, sector 16596472
[ 5660.227290] end_request: I/O error, dev hda, sector 16596592
[ 5660.227299] end_request: I/O error, dev hda, sector 16596784
[ 5660.227308] end_request: I/O error, dev hda, sector 16596896
[ 5660.227326] end_request: I/O error, dev hda, sector 16597104
[ 5660.227336] end_request: I/O error, dev hda, sector 16597400
[ 5660.227345] end_request: I/O error, dev hda, sector 16597752
[ 5660.227354] end_request: I/O error, dev hda, sector 16598296
[ 5660.227371] end_request: I/O error, dev hda, sector 16598784
[ 5660.227381] end_request: I/O error, dev hda, sector 16599160
[ 5660.227390] end_request: I/O error, dev hda, sector 16599408
[ 5660.227399] end_request: I/O error, dev hda, sector 16599728
[ 5660.227416] end_request: I/O error, dev hda, sector 16599968
[ 5660.227426] end_request: I/O error, dev hda, sector 16600184
[ 5660.227436] end_request: I/O error, dev hda, sector 16600672
[ 5660.227445] end_request: I/O error, dev hda, sector 16601064
[ 5660.227470] end_request: I/O error, dev hda, sector 16601664
[ 5660.227480] end_request: I/O error, dev hda, sector 16601960
[ 5660.227489] end_request: I/O error, dev hda, sector 16602104
[ 5660.227498] end_request: I/O error, dev hda, sector 16602432
[ 5660.227515] end_request: I/O error, dev hda, sector 16602728
[ 5660.227525] end_request: I/O error, dev hda, sector 16602824
[ 5660.227534] end_request: I/O error, dev hda, sector 16603264
[ 5660.227543] end_request: I/O error, dev hda, sector 16604072
[ 5660.227559] end_request: I/O error, dev hda, sector 16604344
[ 5660.227569] end_request: I/O error, dev hda, sector 16604864
[ 5660.227578] end_request: I/O error, dev hda, sector 16605528
[ 5660.227587] end_request: I/O error, dev hda, sector 16605704
[ 5660.227610] end_request: I/O error, dev hda, sector 16606096
[ 5660.227620] end_request: I/O error, dev hda, sector 16606656
[ 5660.227630] end_request: I/O error, dev hda, sector 16606832
[ 5660.227639] end_request: I/O error, dev hda, sector 16607000
[ 5660.227656] end_request: I/O error, dev hda, sector 16607280
[ 5660.227665] end_request: I/O error, dev hda, sector 16607648
[ 5660.227674] end_request: I/O error, dev hda, sector 16607824
[ 5660.227683] end_request: I/O error, dev hda, sector 16608312
[ 5660.227700] end_request: I/O error, dev hda, sector 16608936
[ 5660.227710] end_request: I/O error, dev hda, sector 16609920
[ 5660.227719] end_request: I/O error, dev hda, sector 16610072
[ 5660.227728] end_request: I/O error, dev hda, sector 16610304
[ 5660.227745] end_request: I/O error, dev hda, sector 16610552
[ 5660.227755] end_request: I/O error, dev hda, sector 16610808
[ 5660.227764] end_request: I/O error, dev hda, sector 16610920
[ 5660.227773] end_request: I/O error, dev hda, sector 16611072
[ 5660.227796] end_request: I/O error, dev hda, sector 16611224
[ 5660.227806] end_request: I/O error, dev hda, sector 16611392
[ 5660.227815] end_request: I/O error, dev hda, sector 16611544
[ 5660.227824] end_request: I/O error, dev hda, sector 16612560
[ 5660.227841] end_request: I/O error, dev hda, sector 16613128
[ 5660.227851] end_request: I/O error, dev hda, sector 16613400
[ 5660.227860] end_request: I/O error, dev hda, sector 16614016
[ 5660.227869] end_request: I/O error, dev hda, sector 16614664
[ 5660.227886] end_request: I/O error, dev hda, sector 16614760
[ 5660.227896] end_request: I/O error, dev hda, sector 16615048
[ 5660.227905] end_request: I/O error, dev hda, sector 16615216
[ 5660.227914] end_request: I/O error, dev hda, sector 16615416
[ 5660.227940] end_request: I/O error, dev hda, sector 16615824
[ 5660.227950] end_request: I/O error, dev hda, sector 16616216
[ 5660.227959] end_request: I/O error, dev hda, sector 16616352
[ 5660.227968] end_request: I/O error, dev hda, sector 16617576
[ 5660.227984] end_request: I/O error, dev hda, sector 16617904
[ 5660.227994] end_request: I/O error, dev hda, sector 16618000
[ 5660.228001] end_request: I/O error, dev hda, sector 16618368
[ 5660.228676] end_request: I/O error, dev hda, sector 16618816
[ 5660.228696] end_request: I/O error, dev hda, sector 16618912
[ 5660.228706] end_request: I/O error, dev hda, sector 16625328
[ 5660.228715] end_request: I/O error, dev hda, sector 16625568
[ 5660.228724] end_request: I/O error, dev hda, sector 16625816
[ 5660.228741] end_request: I/O error, dev hda, sector 16626144
[ 5660.228751] end_request: I/O error, dev hda, sector 16626768
[ 5660.228760] end_request: I/O error, dev hda, sector 16627568
[ 5660.228769] end_request: I/O error, dev hda, sector 16628976
[ 5660.228793] end_request: I/O error, dev hda, sector 16629488
[ 5660.228802] end_request: I/O error, dev hda, sector 16629856
[ 5660.228832] end_request: I/O error, dev hda, sector 16630128
[ 5660.228841] end_request: I/O error, dev hda, sector 16630624
[ 5660.228859] end_request: I/O error, dev hda, sector 16633464
[ 5660.228869] end_request: I/O error, dev hda, sector 16633656
[ 5660.228878] end_request: I/O error, dev hda, sector 16633880
[ 5660.228887] end_request: I/O error, dev hda, sector 16634288
[ 5660.228904] end_request: I/O error, dev hda, sector 16634600
[ 5660.228913] end_request: I/O error, dev hda, sector 16634744
[ 5660.228922] end_request: I/O error, dev hda, sector 16634856
[ 5660.228931] end_request: I/O error, dev hda, sector 16634952
[ 5660.228955] end_request: I/O error, dev hda, sector 16635096
[ 5660.228965] end_request: I/O error, dev hda, sector 16635144
[ 5660.228974] end_request: I/O error, dev hda, sector 16635472
[ 5660.229002] end_request: I/O error, dev hda, sector 16635744
[ 5660.229020] end_request: I/O error, dev hda, sector 16635912
[ 5660.229030] end_request: I/O error, dev hda, sector 16636176
[ 5660.229039] end_request: I/O error, dev hda, sector 16636360
[ 5660.229048] end_request: I/O error, dev hda, sector 16636544
[ 5660.229065] end_request: I/O error, dev hda, sector 16636688
[ 5660.229074] end_request: I/O error, dev hda, sector 16637008
[ 5660.229084] end_request: I/O error, dev hda, sector 16637208
[ 5660.229093] end_request: I/O error, dev hda, sector 16637376
[ 5660.229133] end_request: I/O error, dev hda, sector 16638128
[ 5660.229143] end_request: I/O error, dev hda, sector 16638384
[ 5660.229170] end_request: I/O error, dev hda, sector 16638536
[ 5660.229180] end_request: I/O error, dev hda, sector 16638984
[ 5660.229226] end_request: I/O error, dev hda, sector 16639176
[ 5660.229236] end_request: I/O error, dev hda, sector 16639336
[ 5660.229245] end_request: I/O error, dev hda, sector 16639688
[ 5660.229254] end_request: I/O error, dev hda, sector 16639816
[ 5660.229272] end_request: I/O error, dev hda, sector 16640088
[ 5660.229282] end_request: I/O error, dev hda, sector 16640832
[ 5660.229291] end_request: I/O error, dev hda, sector 16641336
[ 5660.229300] end_request: I/O error, dev hda, sector 16641840
[ 5660.229354] end_request: I/O error, dev hda, sector 16642792
[ 5660.229364] end_request: I/O error, dev hda, sector 16642960
[ 5660.229373] end_request: I/O error, dev hda, sector 16643304
[ 5660.229382] end_request: I/O error, dev hda, sector 16643512
[ 5660.229435] end_request: I/O error, dev hda, sector 16644568
[ 5660.229446] end_request: I/O error, dev hda, sector 16644576
[ 5660.229455] end_request: I/O error, dev hda, sector 16644584
[ 5660.229464] end_request: I/O error, dev hda, sector 16644592
[ 5660.229473] end_request: I/O error, dev hda, sector 16644600
[ 5660.229482] end_request: I/O error, dev hda, sector 16644608
[ 5660.229491] end_request: I/O error, dev hda, sector 16644616
[ 5660.229499] end_request: I/O error, dev hda, sector 16644624
[ 5660.229509] end_request: I/O error, dev hda, sector 16644632
[ 5660.229517] end_request: I/O error, dev hda, sector 16644640
[ 5660.229526] end_request: I/O error, dev hda, sector 16644648
[ 5660.229535] end_request: I/O error, dev hda, sector 16644656
[ 5660.229544] end_request: I/O error, dev hda, sector 16644664
[ 5660.229553] end_request: I/O error, dev hda, sector 16661240
[ 5660.229562] end_request: I/O error, dev hda, sector 16661248
[ 5660.229571] end_request: I/O error, dev hda, sector 16661264
[ 5660.229580] end_request: I/O error, dev hda, sector 16661272
[ 5660.229589] end_request: I/O error, dev hda, sector 16661280
[ 5660.229598] end_request: I/O error, dev hda, sector 16661288
[ 5660.229607] end_request: I/O error, dev hda, sector 16661296
[ 5660.229615] end_request: I/O error, dev hda, sector 16661304
[ 5660.229624] end_request: I/O error, dev hda, sector 16661312
[ 5660.229633] end_request: I/O error, dev hda, sector 16661320
[ 5660.229642] end_request: I/O error, dev hda, sector 16661328
[ 5660.229651] end_request: I/O error, dev hda, sector 16661336
[ 5660.229660] end_request: I/O error, dev hda, sector 16661344
[ 5660.229668] end_request: I/O error, dev hda, sector 16661352
[ 5660.229677] end_request: I/O error, dev hda, sector 16661360
[ 5660.229686] end_request: I/O error, dev hda, sector 16661368
[ 5660.229694] end_request: I/O error, dev hda, sector 16661376
[ 5660.229703] end_request: I/O error, dev hda, sector 16661392
[ 5660.229724] end_request: I/O error, dev hda, sector 16661408
[ 5660.229734] end_request: I/O error, dev hda, sector 16661440
[ 5660.229743] end_request: I/O error, dev hda, sector 16661448
[ 5660.229752] end_request: I/O error, dev hda, sector 16661456
[ 5660.229761] end_request: I/O error, dev hda, sector 16661464
[ 5660.229770] end_request: I/O error, dev hda, sector 16661480
[ 5660.229779] end_request: I/O error, dev hda, sector 16661496
[ 5660.229798] end_request: I/O error, dev hda, sector 16661504
[ 5660.229807] end_request: I/O error, dev hda, sector 16661512
[ 5660.229817] end_request: I/O error, dev hda, sector 16661528
[ 5660.229826] end_request: I/O error, dev hda, sector 16661544
[ 5660.229834] end_request: I/O error, dev hda, sector 16661552
[ 5660.229843] end_request: I/O error, dev hda, sector 16661568
[ 5660.229876] end_request: I/O error, dev hda, sector 16661576
[ 5660.229886] end_request: I/O error, dev hda, sector 16661592
[ 5660.229895] end_request: I/O error, dev hda, sector 16661600
[ 5660.229904] end_request: I/O error, dev hda, sector 16661608
[ 5660.229913] end_request: I/O error, dev hda, sector 16661616
[ 5660.229922] end_request: I/O error, dev hda, sector 16661624
[ 5660.229931] end_request: I/O error, dev hda, sector 16661632
[ 5660.229939] end_request: I/O error, dev hda, sector 16661640
[ 5660.229948] end_request: I/O error, dev hda, sector 16661648
[ 5660.229957] end_request: I/O error, dev hda, sector 16661664
[ 5660.229966] end_request: I/O error, dev hda, sector 16661672
[ 5660.229975] end_request: I/O error, dev hda, sector 16661680
[ 5660.229984] end_request: I/O error, dev hda, sector 16661688
[ 5660.229993] end_request: I/O error, dev hda, sector 16661696
[ 5660.230002] end_request: I/O error, dev hda, sector 16661704
[ 5660.230010] end_request: I/O error, dev hda, sector 16661712
[ 5660.230019] end_request: I/O error, dev hda, sector 16661728
[ 5660.230038] end_request: I/O error, dev hda, sector 16661736
[ 5660.230048] end_request: I/O error, dev hda, sector 16661752
[ 5660.230056] end_request: I/O error, dev hda, sector 16661760
[ 5660.230065] end_request: I/O error, dev hda, sector 16661776
[ 5660.230074] end_request: I/O error, dev hda, sector 16661808
[ 5660.230109] end_request: I/O error, dev hda, sector 16661832
[ 5660.230119] end_request: I/O error, dev hda, sector 16661840
[ 5660.230128] end_request: I/O error, dev hda, sector 16661848
[ 5660.230136] end_request: I/O error, dev hda, sector 16661856
[ 5660.230145] end_request: I/O error, dev hda, sector 16661864
[ 5660.230154] end_request: I/O error, dev hda, sector 16661872
[ 5660.230163] end_request: I/O error, dev hda, sector 16661912
[ 5660.230172] end_request: I/O error, dev hda, sector 16661920
[ 5660.230181] end_request: I/O error, dev hda, sector 16661928
[ 5660.230190] end_request: I/O error, dev hda, sector 16661936
[ 5660.230198] end_request: I/O error, dev hda, sector 16661944
[ 5660.230207] end_request: I/O error, dev hda, sector 16661952
[ 5660.230216] end_request: I/O error, dev hda, sector 16661960
[ 5660.230225] end_request: I/O error, dev hda, sector 16661976
[ 5660.230234] end_request: I/O error, dev hda, sector 16661984
[ 5660.230242] end_request: I/O error, dev hda, sector 16661992
[ 5660.230251] end_request: I/O error, dev hda, sector 16662000
[ 5660.230260] end_request: I/O error, dev hda, sector 16662008
[ 5660.230269] end_request: I/O error, dev hda, sector 16662024
[ 5660.230320] end_request: I/O error, dev hda, sector 16662032
[ 5660.230330] end_request: I/O error, dev hda, sector 16662040
[ 5660.230339] end_request: I/O error, dev hda, sector 16662048
[ 5660.230348] end_request: I/O error, dev hda, sector 16662056
[ 5660.230357] end_request: I/O error, dev hda, sector 16662064
[ 5660.230365] end_request: I/O error, dev hda, sector 16662072
[ 5660.230374] end_request: I/O error, dev hda, sector 16662080
[ 5660.230383] end_request: I/O error, dev hda, sector 16662088
[ 5660.230391] end_request: I/O error, dev hda, sector 16662096
[ 5660.230401] end_request: I/O error, dev hda, sector 16662112
[ 5660.230409] end_request: I/O error, dev hda, sector 16662120
[ 5660.230419] end_request: I/O error, dev hda, sector 16662136
[ 5660.230427] end_request: I/O error, dev hda, sector 16662144
[ 5660.230436] end_request: I/O error, dev hda, sector 16662168
[ 5660.230458] end_request: I/O error, dev hda, sector 16662176
[ 5660.230468] end_request: I/O error, dev hda, sector 16662184
[ 5660.230477] end_request: I/O error, dev hda, sector 16662208
[ 5660.230486] end_request: I/O error, dev hda, sector 16662216
[ 5660.230495] end_request: I/O error, dev hda, sector 16662232
[ 5660.230504] end_request: I/O error, dev hda, sector 16662240
[ 5660.230513] end_request: I/O error, dev hda, sector 16662256
[ 5660.230541] end_request: I/O error, dev hda, sector 16662280
[ 5660.230551] end_request: I/O error, dev hda, sector 16662288
[ 5660.230560] end_request: I/O error, dev hda, sector 16662320
[ 5660.230569] end_request: I/O error, dev hda, sector 16662328
[ 5660.230578] end_request: I/O error, dev hda, sector 16662336
[ 5660.230587] end_request: I/O error, dev hda, sector 16662344
[ 5660.230596] end_request: I/O error, dev hda, sector 16662352
[ 5660.230604] end_request: I/O error, dev hda, sector 16662360
[ 5660.230613] end_request: I/O error, dev hda, sector 16662368
[ 5660.230622] end_request: I/O error, dev hda, sector 16662376
[ 5660.230631] end_request: I/O error, dev hda, sector 16662384
[ 5660.230640] end_request: I/O error, dev hda, sector 16662400
[ 5660.230648] end_request: I/O error, dev hda, sector 16662408
[ 5660.230657] end_request: I/O error, dev hda, sector 16662432
[ 5660.230678] end_request: I/O error, dev hda, sector 16662440
[ 5660.230687] end_request: I/O error, dev hda, sector 16662448
[ 5660.230696] end_request: I/O error, dev hda, sector 16662456
[ 5660.230705] end_request: I/O error, dev hda, sector 16662464
[ 5660.230714] end_request: I/O error, dev hda, sector 16662480
[ 5660.230723] end_request: I/O error, dev hda, sector 16662496
[ 5660.230732] end_request: I/O error, dev hda, sector 16662528
[ 5660.230752] end_request: I/O error, dev hda, sector 16662536
[ 5660.230762] end_request: I/O error, dev hda, sector 16662552
[ 5660.230771] end_request: I/O error, dev hda, sector 16662560
[ 5660.230780] end_request: I/O error, dev hda, sector 16662584
[ 5660.230789] end_request: I/O error, dev hda, sector 16662624
[ 5660.230808] end_request: I/O error, dev hda, sector 16662632
[ 5660.230817] end_request: I/O error, dev hda, sector 16662640
[ 5660.230827] end_request: I/O error, dev hda, sector 16662656
[ 5660.230835] end_request: I/O error, dev hda, sector 16662664
[ 5660.230844] end_request: I/O error, dev hda, sector 16662680
[ 5660.230853] end_request: I/O error, dev hda, sector 16662696
[ 5660.230897] end_request: I/O error, dev hda, sector 16662712
[ 5660.230907] end_request: I/O error, dev hda, sector 16662784
[ 5660.230916] end_request: I/O error, dev hda, sector 16662792
[ 5660.230924] end_request: I/O error, dev hda, sector 16662800
[ 5660.230933] end_request: I/O error, dev hda, sector 16662816
[ 5660.230942] end_request: I/O error, dev hda, sector 16662824
[ 5660.230951] end_request: I/O error, dev hda, sector 16662840
[ 5660.230974] end_request: I/O error, dev hda, sector 16662880
[ 5660.230984] end_request: I/O error, dev hda, sector 16662896
[ 5660.230993] end_request: I/O error, dev hda, sector 16662904
[ 5660.231002] end_request: I/O error, dev hda, sector 16662936
[ 5660.231011] end_request: I/O error, dev hda, sector 16662944
[ 5660.231020] end_request: I/O error, dev hda, sector 16662952
[ 5660.231029] end_request: I/O error, dev hda, sector 16662960
[ 5660.231038] end_request: I/O error, dev hda, sector 16662968
[ 5660.231046] end_request: I/O error, dev hda, sector 16662976
[ 5660.231055] end_request: I/O error, dev hda, sector 16662992
[ 5660.231080] end_request: I/O error, dev hda, sector 16663000
[ 5660.231090] end_request: I/O error, dev hda, sector 16663008
[ 5660.231098] end_request: I/O error, dev hda, sector 16663016
[ 5660.231107] end_request: I/O error, dev hda, sector 16663024
[ 5660.231116] end_request: I/O error, dev hda, sector 16663048
[ 5660.231125] end_request: I/O error, dev hda, sector 16663056
[ 5660.231134] end_request: I/O error, dev hda, sector 16663072
[ 5660.231143] end_request: I/O error, dev hda, sector 16663080
[ 5660.231152] end_request: I/O error, dev hda, sector 16663088
[ 5660.231161] end_request: I/O error, dev hda, sector 16663104
[ 5660.231184] end_request: I/O error, dev hda, sector 16663144
[ 5660.231198] end_request: I/O error, dev hda, sector 16663152
[ 5660.231225] end_request: I/O error, dev hda, sector 16663160
[ 5660.231270] end_request: I/O error, dev hda, sector 16663168
[ 5660.231279] end_request: I/O error, dev hda, sector 16663176
[ 5660.231289] end_request: I/O error, dev hda, sector 16663192
[ 5660.231298] end_request: I/O error, dev hda, sector 16663208
[ 5660.231307] end_request: I/O error, dev hda, sector 16663224
[ 5660.231330] end_request: I/O error, dev hda, sector 16663232
[ 5660.231340] end_request: I/O error, dev hda, sector 16663240
[ 5660.231377] end_request: I/O error, dev hda, sector 16663256
[ 5660.231419] end_request: I/O error, dev hda, sector 16663264
[ 5660.231429] end_request: I/O error, dev hda, sector 16663288
[ 5660.231438] end_request: I/O error, dev hda, sector 16663304
[ 5660.231456] end_request: I/O error, dev hda, sector 16663312
[ 5660.231466] end_request: I/O error, dev hda, sector 16663504
[ 5660.231475] end_request: I/O error, dev hda, sector 16663520
[ 5660.231485] end_request: I/O error, dev hda, sector 16663536
[ 5660.231586] end_request: I/O error, dev hda, sector 16663552
[ 5660.231596] end_request: I/O error, dev hda, sector 16663560
[ 5660.231605] end_request: I/O error, dev hda, sector 16663568
[ 5660.231614] end_request: I/O error, dev hda, sector 16663576
[ 5660.231623] end_request: I/O error, dev hda, sector 16663976
[ 5660.231632] end_request: I/O error, dev hda, sector 16663984
[ 5660.231641] end_request: I/O error, dev hda, sector 16664008
[ 5660.231650] end_request: I/O error, dev hda, sector 16664024
[ 5660.231668] end_request: I/O error, dev hda, sector 16664032
[ 5660.231677] end_request: I/O error, dev hda, sector 16664096
[ 5660.231686] end_request: I/O error, dev hda, sector 16664160
[ 5660.231695] end_request: I/O error, dev hda, sector 16664184
[ 5660.231721] end_request: I/O error, dev hda, sector 16664200
[ 5660.231730] end_request: I/O error, dev hda, sector 16664208
[ 5660.231739] end_request: I/O error, dev hda, sector 16664232
[ 5660.231748] end_request: I/O error, dev hda, sector 16664240
[ 5660.231757] end_request: I/O error, dev hda, sector 16664272
[ 5660.231766] end_request: I/O error, dev hda, sector 16664280
[ 5660.231775] end_request: I/O error, dev hda, sector 16664288
[ 5660.231954] end_request: I/O error, dev hda, sector 16664296
[ 5660.231964] end_request: I/O error, dev hda, sector 16664312
[ 5661.254187] end_request: I/O error, dev hda, sector 146160
[ 5661.254212] end_request: I/O error, dev hda, sector 146168
[ 5661.254222] end_request: I/O error, dev hda, sector 150240
[ 5661.254230] end_request: I/O error, dev hda, sector 150272
[ 5661.254237] end_request: I/O error, dev hda, sector 150288
[ 5661.254501] end_request: I/O error, dev hda, sector 150296
[ 5661.254521] end_request: I/O error, dev hda, sector 150304
[ 5661.254529] end_request: I/O error, dev hda, sector 150312
[ 5661.254536] end_request: I/O error, dev hda, sector 150328
[ 5661.254544] end_request: I/O error, dev hda, sector 150336
[ 5661.254551] end_request: I/O error, dev hda, sector 150352
[ 5661.254559] end_request: I/O error, dev hda, sector 150360
[ 5661.254566] end_request: I/O error, dev hda, sector 150368
[ 5661.254573] end_request: I/O error, dev hda, sector 150376
[ 5661.254581] end_request: I/O error, dev hda, sector 150392
[ 5661.254817] end_request: I/O error, dev hda, sector 150408
[ 5661.254836] end_request: I/O error, dev hda, sector 150424
[ 5661.254844] end_request: I/O error, dev hda, sector 150432
[ 5661.254851] end_request: I/O error, dev hda, sector 150440
[ 5661.254859] end_request: I/O error, dev hda, sector 150448
[ 5661.254866] end_request: I/O error, dev hda, sector 150456
[ 5661.254873] end_request: I/O error, dev hda, sector 150464
[ 5661.254881] end_request: I/O error, dev hda, sector 150472
[ 5661.255037] end_request: I/O error, dev hda, sector 150488
[ 5661.255114] end_request: I/O error, dev hda, sector 150496
[ 5661.255122] end_request: I/O error, dev hda, sector 150504
[ 5661.255129] end_request: I/O error, dev hda, sector 150512
[ 5661.255136] end_request: I/O error, dev hda, sector 150520
[ 5661.255144] end_request: I/O error, dev hda, sector 150536
[ 5661.255618] end_request: I/O error, dev hda, sector 150544
[ 5661.255640] end_request: I/O error, dev hda, sector 150552
[ 5661.255640] end_request: I/O error, dev hda, sector 150560
[ 5661.255640] end_request: I/O error, dev hda, sector 150568
[ 5661.255640] end_request: I/O error, dev hda, sector 150576
[ 5661.255640] end_request: I/O error, dev hda, sector 150584
[ 5661.255640] end_request: I/O error, dev hda, sector 150592
[ 5661.255640] end_request: I/O error, dev hda, sector 150616
[ 5661.255640] end_request: I/O error, dev hda, sector 150632
[ 5661.255640] end_request: I/O error, dev hda, sector 150640
[ 5661.255640] end_request: I/O error, dev hda, sector 150648
[ 5661.255640] end_request: I/O error, dev hda, sector 150656
[ 5661.255640] end_request: I/O error, dev hda, sector 150664
[ 5661.255640] end_request: I/O error, dev hda, sector 150672
[ 5661.255640] end_request: I/O error, dev hda, sector 150680
[ 5661.255640] end_request: I/O error, dev hda, sector 150688
[ 5661.255640] end_request: I/O error, dev hda, sector 150696
[ 5661.255640] end_request: I/O error, dev hda, sector 150704
[ 5661.255640] end_request: I/O error, dev hda, sector 150720
[ 5661.255640] end_request: I/O error, dev hda, sector 150728
[ 5661.255661] end_request: I/O error, dev hda, sector 154448
[ 5661.255669] end_request: I/O error, dev hda, sector 154664
[ 5661.255676] end_request: I/O error, dev hda, sector 155024
[ 5661.255691] end_request: I/O error, dev hda, sector 155168
[ 5661.255721] end_request: I/O error, dev hda, sector 155376
[ 5661.255728] end_request: I/O error, dev hda, sector 155528
[ 5661.255735] end_request: I/O error, dev hda, sector 155704

Re: I/O Error with heavy I/O

Posted: 18. Oct 2010, 11:13
by econtreras
Hi

i forgot this dump :

[ 6355.854696] end_request: I/O error, dev hda, sector 13293384
[ 6374.799966] end_request: I/O error, dev hda, sector 10933432
[ 6423.540041] end_request: I/O error, dev hda, sector 10933432
[ 6423.652575] sf_read_super_aux err=-71
[ 6423.653088] kobject_add_internal failed for vboxsf-working with -EEXIST, don't try to register things with the same name in the same directory.
[ 6423.653104] Pid: 2731, comm: mount.vboxsf Tainted: G W 2.6.26-1-amd64 #1
[ 6423.653115]
[ 6423.653123] Call Trace:
[ 6423.653249] [<ffffffff8031b305>] kobject_add_internal+0x13f/0x17e
[ 6423.653249] [<ffffffff8031b742>] kobject_add+0x74/0x7c
[ 6423.653249] [<ffffffffa0142975>] :vboxguest:VBoxGuestCommonIOCtl+0x4c3/0x1674
[ 6423.653249] [<ffffffff8031b15c>] kobject_get+0x12/0x17
[ 6423.653249] [<ffffffff80382b41>] get_device+0x17/0x20
[ 6423.653249] [<ffffffff80383151>] device_add+0x9b/0x53f
[ 6423.653249] [<ffffffff8031afc6>] kobject_init+0x41/0x69
[ 6423.653249] [<ffffffff803836a1>] device_create_vargs+0x9a/0xc6
[ 6423.653249] [<ffffffff8027d40f>] bdi_register+0x57/0xb4
[ 6423.653249] [<ffffffff80323a44>] percpu_counter_init+0x6c/0x72
[ 6423.653249] [<ffffffff8027d313>] bdi_init+0x7d/0xab
[ 6423.653249] [<ffffffffa0232c78>] :vboxsf:sf_init_backing_dev+0x24/0x44
[ 6423.653249] [<ffffffffa02314ce>] :vboxsf:sf_read_super_26+0x420/0x4d6
[ 6423.653249] [<ffffffff8029cff6>] set_anon_super+0x0/0xa3
[ 6423.653249] [<ffffffffa02310ae>] :vboxsf:sf_read_super_26+0x0/0x4d6
[ 6423.653249] [<ffffffff8029d22d>] get_sb_nodev+0x4f/0x97
[ 6423.653249] [<ffffffff8029cbbc>] vfs_kern_mount+0x93/0x11b
[ 6423.653249] [<ffffffff8029cc97>] do_kern_mount+0x43/0xdc
[ 6423.653249] [<ffffffff802b16b1>] do_new_mount+0x5b/0x95
[ 6423.653249] [<ffffffff802b18a8>] do_mount+0x1bd/0x1e7
[ 6423.653249] [<ffffffff80276856>] __alloc_pages_internal+0xd6/0x3bf
[ 6423.653249] [<ffffffff802b195c>] sys_mount+0x8a/0xce
[ 6423.653249] [<ffffffff8020beca>] system_call_after_swapgs+0x8a/0x8f
[ 6423.653249]
[ 6423.654601] sf_read_super_aux err=-22
[ 6423.676126] sf_read_super_aux err=-71
[ 6423.682701] sf_read_super_aux err=-71
[ 6423.683067] sf_read_super_aux err=-71

Thanks a lot
Eduardo