Can't boot: “mdadm: CREATE group disk not found”

Question

After running apt-get upgrade yesterday and shutting down, I'm no longer able to boot Ubuntu 15.

The last several lines I see during boot are:

Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.

My hard disk is encrypted, so normally I would get a Passphrase prompt here. However, after a little while, I just get:

mdadm: CREATE group disk not found

This message repeats ad infinitum and I'm not able to boot. Same with recovery mode.

I found a few threads about this error, but no solution yet. What can I do to diagnose and fix this problem?

Thank you!

Answer 1

This error is generated by the initramfs script that finds and mounts the root fs.

I've experienced this message before and it turned out to be caused by updating the initramfs from a live cd.

The thing was that I had an encrypted root, and when opening the luks volume from the live cd I didn't use the same name that was in my crypttab.

So, I booted the live cd and did:

cryptsetup luksOpen /dev/md1 md1
vgchange -ay
mount /dev/vgmain/lvroot /mnt/custom
<... mount dev et. al ...>
chroot /mnt/custom
<... fix something ...>
update-initramfs -u
exit
umount /mnt/custom && vgchange -an && cryptsetup luksClose md1

The update-initramfs -u did warn about cryptsetup: WARNING: invalid line in /etc/crypttab - but at first I didn't deem it important.

After some failed attempts to boot it hit me: my crypttab had md1_crypt for the luks volume, but when I updated the initramfs it saw md1 so it went with that. From within the boot scripts md1 was not available as a luks volume.

So I booted my livecd again and corrected it:

cryptsetup luksOpen /dev/md1 md1_crypt
vgchange -ay
mount /dev/vgmain/lvroot /mnt/custom
<... mount dev et. al ...>
chroot /mnt/custom
update-initramfs -u
exit
umount /mnt/custom
vgchange -an
cryptsetup luksClose md1_crypt

I've looked through the initramfs-tools scripts, but I couldn't find the exact spot where this went wrong, so I assume it was just some weird interaction between cryptsetup, mdadm and lvm.

On another debian host I had a similar issue*, except this time without crypttools or lvm involved, and I was able to work around it by changing my mdadm.conf from /dev/md/n device paths to /dev/mdn. On this ocassion, the issue only presented itself while the array was rebuilding and not when everything was normal.

Perhaps someone more familiar with the inner workings of initramfs-tools can figure this out.

_{* the debian host also showed a message like incrementally starting raid arrays after a few CREATE group disk not found.}

Answer 2

I believe it has to do with race conditions when mdadm builds disk arrays. You may have to play with delays in the pre-mount scripts to allow all the arrays to be built. I have set up a nested RAID for /boot because grub cannot deal with RAID10 and the second level mirror RAID1 does not get normally built by mdadm. So I have had to put mdadm --assemble /dev/md/boot into a pre-mount script.

Your situation maybe different but I think my experience gives a clue.

Answer 3

gnp's answer was pretty good and definitely pointed me in the right direction (my problem was caused by a broken cryptroot setup). However, it wasn't quite sufficient to debug the problem)

1. This can be due to cryptsetup not mapping encrypted volume. This means that initramfs does not have and lvm volume to mount, if you are using and encrypted lvm physical group (the default encrypted ubuntu install)
2. For generally debugging this sort of problem, you want to understand initramfs and update-initramfs (google, then I would insert links by stackexchange won't let me)
3. To get more debugging output you can edit your boot options in grub (by pressing e over a valid option), you should remove the splash and quiet options. If this is not sufficient you can use break=premount as described here: https://wiki.debian.org/InitramfsDebug, to drop into an interactive shell and see what is going on. The script that was causing problems here was called cryptroot.
4. To understand what is going on in the generation of initramfs scripts, you can look at the files in /usr/share/initramfs-tools/hooks and /usr/share/initramfs-tools/scripts, the former is run on a full working system to generate configuration for the latter. You might try adding set -x flags to these scripts for debugging.

In my particular case, I had problems correcting this from a boot disk with chroot because lvm tools where not returning the correct results inside the jail. I needed to mount in a number of things to work around this. I also had to ensure that the luks volume was mapped with the correct name (It needed to match up with the value in crypttab, and the helpful graphical interface was picking quite a bad name.

cryptsetup luksOpen /dev/md1 md1_crypt
mount --bind /sys /mnt/encrypted-root/sys
mount --bind /dev /mnt/encrypted-root/dev
mount --bind /proc /mnt/encrypted-root/proc
chroot /mnt/encrypted-root
update-inintramfs -c -k all