After a bit of investigating of a problem I've just encountered, it looks like mdadm has completely forgotten about a new device I'd added to an array. I recently added a new device to the array which went fine, but I've now rebooted (I don't think I had rebooted since making this change) and I've been greeted with all sorts of errors, and I'm fairly sure it's down to this.
So my question is: Should it be safe to just re-add that device back into the array? I really don't want to lose any data here! This is what I think I should do but I'd like confirmation from someone with a bit more experience first :) Bear in mind I've already done this once, and have fscked and grown the filesystem too, and have then written plenty more data to it.
sudo umount /dev/md0
sudo mdadm --stop /dev/md0
sudo mdadm --add /dev/md0 /dev/sdd1
sudo mdadm --grow /dev/md0 --raid-devices=4
A bunch of extra information and steps I've already taken: I rebooted and was greeted with a message along the lines of "Unable to read superblock" for /dev/md0, my RAID array. I immediately suspected mdadm since I had recently been fiddling with it, and cat /proc/mdstat reports this:
...
md0 : inactive sdb1[0](S) sdc1[1](S)[/code]
Which is wrong; I had just added /dev/sdd1 to the array too. sudo mount -a reported bad superblock on /dev/md0. I can't remember the next command I used now but something else reported that I had tried to write past the end of the filesystem, which makes sense since there was more data on it than could fit onto the first two devices alone.
sudo fdisk -l reports that the new device and partition are alive and well. cat /etc/mdadm/mdadm.conf shows this:
DEVICE /dev/sdb1 /dev/sdc1
ARRAY /dev/md0 level=raid5 num-devices=2 UUID=*snip*
spares=1
Which shows clearly that it's completely forgotten about the new device I added. Was I supposed to 'save' the config after executing the commands above? I assumed they were lasting changes; I can't see any use in them being lost on restarting.
Edit note: I realised that the 'num-devices=2' is inclusive of the 'spares=1', which means that mdadm hasn't changed a disk from 'in use' to 'spare', it's just plain forgotten about it. I'll update the title to account for that.
