Guests going to read-only fs's on 4.0 and 4.1

Discussions related to using VirtualBox on Linux hosts.

Guests going to read-only fs's on 4.0 and 4.1

Postby Spyplane » 7. Sep 2011, 19:14

I have 4 VM boxes in a datacenter which host a number of VM's. The hosts are all Ubuntu server 10.04 machines. The virtualbox versions tried were 4.0 and 4.1 (the latest of each). The guests were: fedora 13 32&64, centos 5.5 32&64, ubuntu 10.04 32&64.

A few days ago a few of the guests starting getting ext4 inode errors and all remounted read-only. I searched around for the issue, but nothing obvious stuck out. I tried upgrading to virtualbox 4.1, but I ran into bug where snapshots didn't complete and when that happens you are in a world of hurt recovering those (lots of .xml hand-editing). So I reverted back down to 4.0 and moved or rebuilt the vm's on a different partition. The partition they were all on were raid 5's and the raid wasn't reporting errors. Everything was rebuilt and worked yesterday and today, 5 vm's are back to being read-only. Now searching around i seem to be the only one hitting this problem so hard. I'm either really unlucky or I'm missing something in my searches. At first it was easy for me to blame the hardware, but now it is so unlikely that 4 separate machines went bad all around the same time.

So, is there anything that is known with Linux hosts + Linux guests that would cause such horrible disk errors?

Thanks for any help you can give, I'm about to the end of my rope on this issue! :)
Spyplane
 
Posts: 2
Joined: 7. Sep 2011, 18:56
Primary OS: Ubuntu other
VBox Version: OSE Debian
Guest OSses: ubuntu, fedora, solaris, centos, open solaris

Re: Guests going to read-only fs's on 4.0 and 4.1

Postby Spyplane » 7. Sep 2011, 19:27

On the HOSTS, I get these type of messages in dmesg:


[7671542.157708] EXT4-fs (dm-2): Unaligned AIO/DIO on inode 655370 by VBoxHeadless; performance will be poor.
[7680241.081434] EXT4-fs (dm-2): Unaligned AIO/DIO on inode 7733261 by VBoxHeadless; performance will be poor.
[7688987.376976] EXT4-fs (dm-2): Unaligned AIO/DIO on inode 655370 by VBoxHeadless; performance will be poor.
[7697659.511615] EXT4-fs (dm-2): Unaligned AIO/DIO on inode 7733261 by VBoxHeadless; performance will be poor.
[7706387.587069] EXT4-fs (dm-2): Unaligned AIO/DIO on inode 655370 by VBoxHeadless; performance will be poor.
[7715053.687722] EXT4-fs (dm-2): Unaligned AIO/DIO on inode 655370 by VBoxHeadless; performance will be poor.
[7723727.794723] EXT4-fs (dm-2): Unaligned AIO/DIO on inode 655370 by VBoxHeadless; performance will be poor.
[7732439.894578] EXT4-fs (dm-2): Unaligned AIO/DIO on inode 655370 by VBoxHeadless; performance will be poor.


Do you know if that is VBox reporting issues on the guests, or the host? It shows up on the host dmesg, but it seems like a vbox error, so I'm thinking it is referring to the guest os fs.
Spyplane
 
Posts: 2
Joined: 7. Sep 2011, 18:56
Primary OS: Ubuntu other
VBox Version: OSE Debian
Guest OSses: ubuntu, fedora, solaris, centos, open solaris

Re: Guests going to read-only fs's on 4.0 and 4.1

Postby Ene Dene » 5. Dec 2011, 14:17

Same thing here:
Code: Select all   Expand viewCollapse view
Dec  5 12:20:12 killers kernel: [383959.621308] EXT4-fs (sda5): Unaligned AIO/DIO on inode 36438058 by VBoxHeadless; performance will be poor.


host: Ubuntu server x64
guest: Xubuntu desktop x64
Ene Dene
 
Posts: 23
Joined: 6. Nov 2008, 00:30
Primary OS: Ubuntu other
VBox Version: PUEL
Guest OSses: Debian, pfsense, Ubuntu

Re: Guests going to read-only fs's on 4.0 and 4.1

Postby gorostas » 6. Mar 2012, 10:36

Oh my good, this is like I'm reading my problem!

I have same problem like you.. been all over net and no luck!

Can someone shade some light on this strange fenomena.. cose my hard disks are ok for sure, and I have good and stable power suply for hardware!

For some strange reason I was pretty sure not good power supply will solve this isue, but I was wrong cose it happend again..
gorostas
 
Posts: 5
Joined: 18. Jun 2010, 09:38
Primary OS: Ubuntu other
VBox Version: OSE Debian
Guest OSses: XP, MAC

Re: Guests going to read-only fs's on 4.0 and 4.1

Postby aeichner » 6. Mar 2012, 12:18

Hi,

the messages on the host have nothing to do with the problem in the guest. The message usually means that some I/O of the guest was not aligned to the block size of the underlying filesystem. This can degrade I/O performance but does not cause any data corruption. A VBox.log and a kernel log from the guest would help to investigate the problem. Opening
a ticket in our public bugtracker and attaching the files there could speed up the process because the bugtracker is checked much more frequently than the forum.
aeichner
Oracle Corporation
 
Posts: 89
Joined: 31. Aug 2007, 19:12

Re: Guests going to read-only fs's on 4.0 and 4.1

Postby vmoral » 30. Mar 2013, 10:08

Hi there.
I was on the same downward spiral and finally found out a solution :

Scenario :
- host : OSuSE 12.3 - 64 bit
- vm's : 1 linux kernel 2.6, 32 bit, 2 linux ker 3 32 bit.
- vm's with ext3 or ext4 as FS.
- host with md raid.

Problem :
After many "Unaligned AIO/DIO".. by "VBoxHeadless" messages the host hung forcing reboot.
That happens randomly with 1, 2 or 3 machines running in parallel.

Solution (part 1) :

The problem presumably arises when the guest OS try to access the disk block device with a different block size than the host.
My host system has a raid with 4096 block size, but ext4 in my host is looking for 512 sized blocks. So, there are cases that the guest block request is not aligned with the host's. ( and considering that the virtual disk is a file and has a VBox specific header... that messes all ).
As Virtualbox "does not (and never did) any kind of re-buffering of read/write requests issued by the guest to guarantee alignment." i tried with the host o/s cache.
In the VM parameters, on the disk controller, enable "host i/o cache".
Now, the disk requests go through the i/o cache manager of the host OS, providing good alignment an a great performance if running several machines on the same board. The Host cache will act as a network storage cache.

More Virtualbox configs (perhaps not relevant for you..):
- Disabled IOAPIC to avoid asyncs.
- Enabled "VT-x/AMD-V" and "Nested Paging" and "PAE"

Plus OS config:
At the same time i've had many problems with the guest's cpu's losing sync.
I supposed that the problem may arise with energy management that the kernel does (stepping clocks, sleep mode...) to ensure a low power consumption... but now they're virtualised...so i disabled these functions adding these kernel parameters to all the VM's :
apm=off powersaved=off nohz=off

After these changes, i could finally start the vm's and now with 4 months uptime.

Hope this helps.

vm
vmoral
 
Posts: 2
Joined: 30. Mar 2013, 09:11


Return to VirtualBox on Linux Hosts

Who is online

Users browsing this forum: MopedTobias and 36 guests