Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I'm hitting an access error with part of a raid array and need help in troubleshooting this.

History: Several raid partitions live on 4 disks. Four days ago some ticking noises were heard from the workstation, and the GUI ubuntu disk utility showed some bad sectors, but things were otherwise green. Yesterday (Thurs, April 17) we got hit with a power failure and a hard reboot. Following the hard reboot the system comes up and mounts most of the raid partitions, but one large critical one (containing /home) is throwing input/output errors.

bpbrown@eguzki:/$ ls home
ls: cannot access home: Input/output error
bpbrown@eguzki:/$ 

We're in Ubuntu 12.04, and owing to the loss of /home, we're at command line only.

On reboot, mdadm showed that the array was resyncing; that seems to have completed but still no luck in accessing /home. Here are the results of mdadm:

bpbrown@eguzki:/$ sudo mdadm -D /dev/md10
/dev/md10:
    Version : 0.90
  Creation Time : Thu Feb  4 16:49:43 2010
   Raid Level : raid5
 Array Size : 2868879360 (2735.98 GiB 2937.73 GB)
Used Dev Size : 956293120 (911.99 GiB 979.24 GB)
Raid Devices : 4
 Total Devices : 4
Preferred Minor : 10
Persistence : Superblock is persistent

Update Time : Fri Apr 19 10:03:46 2013
      State : clean  
 Active Devices : 4 
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

     Layout : left-symmetric
 Chunk Size : 64K

       UUID : 317df11d:4e2edc70:fa3efedc:498284d3
     Events : 0.2121101

Number   Major   Minor   RaidDevice State
   0       8       10        0      active sync   /dev/sda10
   1       8       26        1      active sync   /dev/sdb10
   2       8       42        2      active sync   /dev/sdc10
   3       8       58        3      active sync   /dev/sdd10
bpbrown@eguzki:/$ 

and here's mdstat:

bpbrown@eguzki:/$ cat /proc/mdstat 
Personalities : [raid1] [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid10] 
md1 : active raid1 sda1[0] sdb1[1] sdc1[2] sdd1[3]
      497856 blocks [4/4] [UUUU]

md8 : active raid5 sda8[0] sdb8[1] sdc8[2] sdd8[3]
      5301120 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md6 : active raid5 sda6[0] sdb6[1] sdc6[2] sdd6[3]
      20530752 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md7 : active raid5 sda7[0] sdc7[2] sdd7[3] sdb7[1]
      5301120 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md5 : active raid5 sda5[0] sdd5[3] sdc5[2] sdb5[1]
      5301120 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md10 : active raid5 sda10[0] sdc10[2] sdd10[3] sdb10[1]
      2868879360 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>
bpbrown@eguzki:/$ 

Unmounting and remounting /dev/md10 doesn't seem to help, though I may well have missed a step in correctly mounting the raid array.

If helpful, here's the contents of /etc/fstab:

bpbrown@eguzki:/$ more /etc/fstab
# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
/dev/md5    /               reiserfs relatime        0       1
/dev/md1    /boot           reiserfs notail,relatime 0       2
/dev/md10   /home           xfs      relatime        0       2
/dev/md8    /tmp            reiserfs relatime        0       2
/dev/md6    /usr            reiserfs relatime        0       2
/dev/md7    /var            reiserfs relatime        0       2
/dev/sda9   none            swap    pri=1           0       0
/dev/sdb9   none            swap    pri=1           0       0
/dev/sdc9   none            swap    pri=1           0       0
/dev/sdd9   none            swap    pri=1           0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto,exec,utf8 0       0
bpbrown@eguzki:/$ 

Update April 23: Tried mounting the filesystem directly again and got a possibly useful error message. Here's a short version, omitting some of the call trace:

bpbrown@eguzki:/$dmesg | tail
[  788.335968] XFS (md10): Mounting Filesystem
[  788.516845] XFS (md10): Starting recovery (logdev: internal)
[  790.082900] XFS: Internal error XFS_WANT_CORRUPTED_GOTO at line 1503 of file /build/buildd/linux-3.2.0/fs/xfs/xfs_alloc.c.  Caller 0xffffffffa0226837
[  790.082905] 
[  790.083004] Pid: 3211, comm: mount Tainted: P           O 3.2.0-38-generic #61-Ubuntu
[  790.083010] Call Trace:
   <omitted for brevity>
[  790.084139] XFS (md10): xfs_do_force_shutdown(0x8) called from line 3729 of file /build/buildd/linux-3.2.0/fs/xfs/xfs_bmap.c.  Return address = 0xffffffffa0236e52
[  790.217602] XFS (md10): Corruption of in-memory data detected.  Shutting down filesystem
[  790.217654] XFS (md10): Please umount the filesystem and rectify the problem(s)
[  790.217761] XFS (md10): xfs_imap_to_bp: xfs_trans_read_buf() returned error 5.
[  790.217775] XFS (md10): xlog_recover_clear_agi_bucket: failed to clear agi 5. Continuing.
<last 2 lines repeat 8 times>
[  790.388209] XFS (md10): Ending recovery (logdev: internal)
bpbrown@eguzki:/$

Thank you in advance for any suggestions on how to proceed,

--Ben

share|improve this question
add comment

1 Answer

up vote 0 down vote accepted

Turns out, the root problem here was indeed the XFS file system corruption during the unclean powerdown. Worse, the XFS filesystem had an unresolved log file, leading to the following warnings:

bpbrown@eguzki:/$ sudo xfs_check /dev/md10
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

Mounting was still failing, so we proceeded with xfs_repair -L. This worked quickly (less than 5 minutes) and despite the dire warnings the /home partition was mountable and readable immediately afterwards.

bpbrown@eguzki:/$ sudo xfs_repair -L /dev/md10
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
agi unlinked bucket 34 is 50978 in ag 1 (inode=536921890)
<...>
Phase 7 - verify and correct link counts...
resetting inode 97329 nlinks from 2 to 3
resetting inode 536921890 nlinks from 0 to 2
done
bpbrown@eguzki:/$

As far as we can tell, the system is functional and didn't suffer any critical data loss.

Cray ended up having some useful documentation on xfs_check and xfs_repair for someone new to those tools like myself, so I've included their link in case anyone else runs into these problems for the first time:

http://docs.cray.com/books/S-2377-22/html-S-2377-22/z1029470303.html

Cheers, and thanks to all who were reading this and thinking of ideas,

--Ben

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.