2

I have a physical machine running Ubuntu 16.04 with a zfs root file system, installed following the instructions at https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.04-Root-on-ZFS

I installed the bootloader as per instructions:

5.5b For UEFI booting, install GRUB:

grub-install --target=x86_64-efi --efi-directory=/boot/efi \
      --bootloader-id=ubuntu --recheck --no-floppy

The system has been running fine for three months, with several reboots (mainly caused by power cuts).

The root pool is a three way mirror:

NAME                                                                STATE     READ WRITE CKSUM
rpool                                                               ONLINE       0     0     0
  mirror-0                                                          ONLINE       0     0     0
    ata-SAMSUNG_HM500JI_S1WFJ90S818624-part1                        ONLINE       0     0     0
    ata-ST3250820AS_5QE5BVW5-part1                                  ONLINE       0     0     0
    ata-GB0250C8045_9SF0R2RD-part1                                  ONLINE       0     0     0

Originally I also had a hot-spare included in the pool. This morning, I needed the hot spare for use in another server which had a failed device, so I used zpool remove to remove the hot spare from the system, powered down, and physically removed the spare.

Now the server won't boot.

I've tried selecting all the three remaining physical disks to boot from but no joy.

At present, I have rebooted using a live CD. Following the early steps in the installation instructions above, I can see all the zfs pools, so the data is all there. I think the next step might be to chroot into this zpool but am not sure how to do so given the different ROOT file systems etc.

I am guessing that the GRUB boot information, for some reason, was only installed onto the disk which was designated as "spare" and which I have now removed. The disk in question is now part of a zfs mirror on a different server so it is not possible to put it back.

My question is: what is the easiest way of re-initialising GRUB so that the server will boot? Do I need to chroot into the disk-based system or can this be done from the live-CD environment? If the former, how do I correctly mount the root pool?

3

Largely cribbed from the ZFS installation instructions at https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.04-Root-on-ZFS, here are the steps I used to get my system working again.

boot-repair did not work.

Step 1: Prepare The Install Environment

1.1 Boot the Ubuntu Live CD, select Try Ubuntu Without Installing, and open a terminal (press Ctrl-Alt-T).

1.2 Optional: Install the OpenSSH server in the Live CD environment: If you have a second system, using SSH to access the target system can be convenient.

$ sudo apt-get --yes install openssh-server

Set a password on the “ubuntu” (Live CD user) account:

$ passwd

Hint: You can find your IP address with ip addr show scope global. Then, from your main machine, connect with ssh ubuntu@IP.

1.3 Become root:

# sudo -i

1.4 Install ZFS in the Live CD environment:

# apt-add-repository universe
# apt update

(ignore errors about moving an old database out of the way)

# apt install --yes debootstrap gdisk zfs-initramfs

Step 2: Discover available ZFS pools

2.1 check if ZFS pools are already imported

# zpool list
# zfs list 

2.2 if so, we need to export the zfs pool so we can mount it in a different directory so we can chroot to it

# zpool export rpool

Step 3: Chroot into ZFS pool

3.1 import pool to non-default location. The -N flag (don’t automatically mount) is necessary because otherwise the rpool root, and the rpool/root/UBUNTU pool, will both try to mount on /mnt

# zpool import -a -N -R /mnt

3.2 mount the root system

# zfs mount rpool/ROOT/ubuntu

3.3 mount the remaining file systems

# zfs mount -a

3.4 Bind the virtual filesystems from the LiveCD environment to the new system and chroot into it:

# mount --rbind /dev  /mnt/dev
# mount --rbind /proc /mnt/proc
# mount --rbind /sys  /mnt/sys
# chroot /mnt /bin/bash --login

Note: This is using --rbind, not —bind.

Step 4: Re-initialise EFI partitions on all root pool components

4.1 Check the wildcard gets the correct root pool partitions:

# for i in /dev/disk/by-id/*ata*part3; do echo $i; done

4.2 Add an entry for /boot/efi for each disk to /etc/fstab for failover purposes in future:

# for i in /dev/disk/by-id/*ata*part3; \
      do mkdosfs -F 32 -n EFI ${i}; \
      echo PARTUUID=$(blkid -s PARTUUID -o value \
      ${i}) /boot/efi vfat defaults 0 1 >> /etc/fstab; done

4.3 mount the first disk

# mount /dev/disk/by-id/scsi-SATA_disk1-part3 /boot/efi

4.4 install grub

# grub-install --target=x86_64-efi --efi-directory=/boot/efi \
      --bootloader-id=ubuntu --recheck —no-floppy

4.5 unmount the first partition

# umount /boot/efi

4.6 mount the second disk

# mount /dev/disk/by-id/scsi-SATA_disk2-part3 /boot/efi

4.7 install grub

# grub-install --target=x86_64-efi --efi-directory=/boot/efi \
      --bootloader-id=ubuntu-2 --recheck —no-floppy

4.8 repeat steps 4.5 to 4.7 for each additional disk 4.9 For added insurance, do an MBR installation to each disk too

# grub-install /dev/disk/by-id/scsi-SATA_disk1
# grub-install /dev/disk/by-id/scsi-SATA_disk2

Step 5: Reboot

5.1 Quit from the chroot

# exit

5.2 Reboot

# reboot
  • Thank you very much for posting this answer! I found myself looking for exactly the same thing after following the walk through. – Nelson Dec 18 '16 at 2:49
  • zfs export rpool should be zpool export I believe – Nelson Dec 18 '16 at 14:03
  • Also, if you get a grub-install: error: /usr/lib/grub/x86_64-efi/modinfo.sh doesn't exist error, it may be that the liveCD environment needs to have grub-efi-amd64 so install with apt install --yes grub-efi-amd64. I did this just after step 4.2, but I would assume it would be ok for an earlier ordering of that step. – Nelson Dec 18 '16 at 20:43
  • Good catch about zfs export! – nickcrabtree Dec 21 '16 at 18:17
0

The easiest and best way of re-initializing grub is a utility called boot repair. Follow these steps:

  1. Boot from live Ubuntu disk and select "try without installing"
  2. Open a terminal and run these commands:

    sudo -i
    add-apt-repository ppa:yannubuntu/boot-repair
    apt update
    apt install boot-repair
    boot-repair
  3. Chose recommended repair

  4. You will eventually be presented with code to paste in the terminal. This will happen twice. Open new terminal window to do this.
  5. After the program finishes restart, removing installation disk.

Boot-repair will completely re-initialize your grub and works well with UEFI.

  • Does boot-repair work with a terminal-only interface and/or RAID? – Kaz Wolfe Sep 17 '16 at 18:23
  • boot-repair user interface works fine from the graphical console from the live disk. Unfortunately, boot-repair fails to repair the system. No code to paste into the terminal was presented. Is boot-repair compatible with a zfs boot partition? – nickcrabtree Sep 17 '16 at 19:28
  • @nickcrabtree recommended repair?? what exactly happens? – Edgy1 Sep 17 '16 at 19:35
  • @KazWolfe he can use just any ubuntu install disk just for the boot repair... – Edgy1 Sep 17 '16 at 19:36
  • Here's the pastebin from boot.repair: paste2.org/fCPC1wFj – nickcrabtree Sep 17 '16 at 20:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.