1

I did the apt-get update and then upgrade to deal with the heartbleed vulnerability. I rebooted after the upgrade without checking that everything was good. Now I'm not having much luck. It is booting and saying that it cannot mount filesystem '-' and putting me in a BusyBox shell.

The good news is that if I boot off of a LiveUSB install, I can follow the instructions and do a zpool import of my rpool. This works. However, I've tried the basics in the Ubuntu ZFS FAQ and I'm still not having much luck.

What steps should I take from the LiveCD in order to get the system going without re-installing?

1

I got things working using the excellent ZFS How-To (This how to is now replaced and newer How-Tos are recommended to be followed)

I booted an Ubuntu 12.04 live usb and used all the commands from step 1 of the HOWTO.

Step 1: Prepare The Install Environment

1.1 Start the Ubuntu LiveCD and open a terminal at the desktop.

1.2 Input these commands at the terminal prompt:

$ sudo -i
# apt-add-repository --yes ppa:zfs-native/stable
# apt-get update
# apt-get install debootstrap spl-dkms zfs-dkms ubuntu-zfs

1.3 Check that the ZFS filesystem is installed and available:

# modprobe zfs
# dmesg | grep ZFS:
ZFS: Loaded module v0.6.3-2~trusty, ZFS pool version 5000, ZFS filesystem version 5

Step 2: Disk Partitioning

This tutorial intentionally recommends MBR partitioning. GPT can be used instead, but beware of UEFI firmware bugs.

2.1 Run your favorite disk partitioner, like parted or cfdisk, on the primary storage device. /dev/disk/by-id/scsi-SATA_disk1 is the example device used in this document.

2.2 Create a small MBR primary partition of at least 8 megabytes. 256mb may be more realistic, unless space is tight. /dev/disk/by-id/scsi-SATA_disk1-part1 is the example boot partition used in this document.

2.3 On this first small partition, set type=BE and enable the bootable flag.

2.4 Create a large partition of at least 4 gigabytes. /dev/disk/by-id/scsi-SATA_disk1-part2 is the example system partition used in this document.

2.5 On this second large partition, set type=BF and disable the bootable flag.

The partition table should look like this:

 # fdisk -l /dev/disk/by-id/scsi-SATA_disk1

 Disk /dev/sda: 10.7 GB, 10737418240 bytes
 255 heads, 63 sectors/track, 1305 cylinders
 Units = cylinders of 16065 * 512 = 8225280 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disk identifier: 0x00000000

 Device    Boot      Start         End      Blocks   Id  System
 /dev/sda1    *          1           1        8001   be  Solaris boot
 /dev/sda2               2        1305    10474380   bf  Solaris

Remember: Substitute scsi-SATA_disk1-part1 and scsi-SATA_disk1-part2 appropriately below.

Hints: * Are you doing this in a virtual machine? Is something in /dev/disk/by-id missing? Go read the troubleshooting section. * Recent GRUB releases assume that the /boot/grub/grubenv file is writable by the stage2 module. Until GRUB gets a ZFS write enhancement, the GRUB modules should be installed to a separate filesystem in a separate partition that is grub-writable. * If /boot/grub is in the ZFS filesystem, then GRUB will fail to boot with this message: error: sparse file not allowed. If you absolutely want only one filesystem, then remove the call to recordfail() in each grub.cfg menu stanza, and edit the /etc/grub.d/10_linux file to make the change permanent. * Alternatively, if /boot/grub is in the ZFS filesystem you can comment each line with the text save_env in the file /etc/grub.d/00_header and run update-grub.

Step 3: Disk Formatting

3.1 Format the small boot partition created by Step 2.2 as a filesystem that has stage1 GRUB support like this:

 # mke2fs -m 0 -L /boot/grub -j /dev/disk/by-id/scsi-SATA_disk1-part1

3.2 Create the root pool on the larger partition:

 # zpool create -o ashift=9 rpool /dev/disk/by-id/scsi-SATA_disk1-part2

Always use the long /dev/disk/by-id/* aliases with ZFS. Using the /dev/sd* device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool.

Warning: The grub2-1.99 package currently published in the PPA for Precise does not reliably handle a 4k block size, which is ashift=12.

Hints: * # ls -la /dev/disk/by-id will list the aliases. * The root pool can be a mirror. For example, zpool create -o ashift=9 rpool mirror /dev/disk/by-id/scsi-SATA_disk1-part2 /dev/disk/by-id/scsi-SATA_disk2-part2. Remember that the version and ashift matter for any pool that GRUB must read, and that these things are difficult to change after pool creation. * If you are using a mirror with a separate boot partition as described above, don't forget to edit the grub.cfg file on the second HD partition so that the "root=" partition refers to that partition on the second HD also; otherwise, if you lose the first disk, you won't be able to boot from the second because the kernel will keep trying to mount the root partition from the first disk. * The pool name is arbitrary. On systems that can automatically install to ZFS, the root pool is named "rpool" by default. Note that system recovery is easier if you choose a unique name instead of "rpool". Anything except "rpool" or "tank", like the hostname, would be a good choice. * If you want to create a mirror but only have one disk available now you can create the mirror using a sparse file as the second member then immediately off-line it so the mirror is in degraded mode. Later you can add another drive to the spool and ZFS will automatically sync them. The sparse file won't take up more than a few KB so it can be bigger than your running system. Just make sure to off-line the sparse file before writing to the pool.

3.2.1 Create a sparse file at least as big as the larger partition on your HDD:

 # truncate -s 11g /tmp/sparsefile

3.2.2 Instead of the command in section 3.2 use this to create the mirror:

 # zpool create -o ashift=9 rpool mirror /dev/disk/by-id/scsi-SATA_disk1-part2 /tmp/sparsefile

3.2.3 Offline the sparse file. You can delete it after this if you want.

 # zpool offline rpool /tmp/sparsefile

3.2.4 Verify that the pool was created and is now degraded.

 # zpool list
 NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
 rpool     10.5G   188K  10.5G     0%  1.00x  DEGRADED  -

3.3 Create a "ROOT" filesystem in the root pool:

 # zfs create rpool/ROOT

3.4 Create a descendant filesystem for the Ubuntu system:

 # zfs create rpool/ROOT/ubuntu-1

On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through pkg image-update or beadm. Similar functionality for APT is possible but currently unimplemented.

3.5 Dismount all ZFS filesystems.

 # zfs umount -a

3.6 Set the mountpoint property on the root filesystem:

 # zfs set mountpoint=/ rpool/ROOT/ubuntu-1

3.7 Set the bootfs property on the root pool.

 # zpool set bootfs=rpool/ROOT/ubuntu-1 rpool

The boot loader uses these two properties to find and start the operating system. These property names are not arbitrary.

Hint: Putting rpool=MyPool or bootfs=MyPool/ROOT/system-1 on the kernel command line overrides the ZFS properties.

3.9 Export the pool:

 # zpool export rpool

Don't skip this step. The system is put into an inconsistent state if this command fails or if you reboot at this point.

(I have include Step 2 and Step 3 for completeness, but I skipped them, since everything was already partitioned and formatted.)

I then ran zpool list to verify that no pools were loaded. (None were) I then did only Step 4.1

zpool import -d /dev/disk/by-id -R /mnt rpool

The full step 4 is below

Step 4: System Installation

Remember: Substitute "rpool" for the name chosen in Step 3.2.

4.1 Import the pool:

 # zpool import -d /dev/disk/by-id -R /mnt rpool

If this fails with "cannot import 'rpool': no such pool available", you can try import the pool without the device name eg:

     # zpool import -R /mnt rpool

4.2 Mount the small boot filesystem for GRUB that was created in step 3.1:

 # mkdir -p /mnt/boot/grub
 # mount /dev/disk/by-id/scsi-SATA_disk1-part1 /mnt/boot/grub

4.4 Install the minimal system:

 # debootstrap trusty /mnt

The debootstrap command leaves the new system in an unconfigured state. In Step 5, we will only do the minimum amount of configuration necessary to make the new system runnable.

I then checked ls -al /mnt to make sure it looked ok.

I then skipped to Step 5.4. It was time to chroot into the imported rpool.

# mount --bind /dev  /mnt/dev
# mount --bind /proc /mnt/proc
# mount --bind /sys  /mnt/sys
# chroot /mnt /bin/bash --login

At this point, we are now in the proper state to fix the system.

The complete step 5 was

Step 5: System Configuration

5.1 Copy these files from the LiveCD environment to the new system:

 # cp /etc/hostname /mnt/etc/
 # cp /etc/hosts /mnt/etc/

5.2 The /mnt/etc/fstab file should be empty except for a comment. Add this line to the /mnt/etc/fstab file:

 /dev/disk/by-id/scsi-SATA_disk1-part1  /boot/grub  auto  defaults  0  1

The regular Ubuntu desktop installer may add dev, proc, sys, or tmp lines to the /etc/fstab file, but such entries are redundant on a system that has a /lib/init/fstab file. Add them now if you want them.

5.3 Edit the /mnt/etc/network/interfaces file so that it contains something like this:

 # interfaces(5) file used by ifup(8) and ifdown(8)
 auto lo
 iface lo inet loopback

 auto eth0
 iface eth0 inet dhcp

Customize this file if the new system is not a DHCP client on the LAN.

5.4 Make virtual filesystems in the LiveCD environment visible to the new system and chroot into it:

 # mount --bind /dev  /mnt/dev
 # mount --bind /proc /mnt/proc
 # mount --bind /sys  /mnt/sys
 # chroot /mnt /bin/bash --login

5.5 Install PPA support in the chroot environment like this:

 # locale-gen en_US.UTF-8
 # apt-get update
 # apt-get install ubuntu-minimal software-properties-common

Even if you prefer a non-English system language, always ensure that en_US.UTF-8 is available. The ubuntu-minimal package is required to use ZoL as packaged in the PPA.

5.6 Install ZFS in the chroot environment for the new system:

 # apt-add-repository --yes ppa:zfs-native/stable
 # apt-add-repository --yes ppa:zfs-native/grub
 # apt-get update
 # apt-get install --no-install-recommends linux-image-generic linux-headers-generic
 # apt-get install ubuntu-zfs
 # apt-get install grub2-common grub-pc
 # apt-get install zfs-initramfs
 # apt-get dist-upgrade

Warning: This is the second time that you must wait for the SPL and ZFS modules to compile. Do not try to skip this step by copying anything from the host environment into the chroot environment.

Note: This should install a kernel package and its headers, a patched mountall and dkms packages. Double-check that you are getting these packages from the PPA if you are deviating from these instructions in any way.

Choose /dev/disk/by-id/scsi-SATA_disk1 if prompted to install the MBR loader.

Ignore warnings that are caused by the chroot environment like:

  • Can not write log, openpty() failed (/dev/pts not mounted?)
  • df: Warning: cannot read table of mounted file systems
  • mtab is not present at /etc/mtab.

    5.7 Set a root password on the new system:

    # passwd root

    Hint: If you want the ubuntu-desktop package, then install it after the first reboot. If you install it now, then it will start several process that must be manually stopped before dismount.

Step 6: GRUB Installation

Remember: All of Step 6 depends on Step 5.4 and must happen inside the chroot environment.

6.1 Verify that the ZFS root filesystem is recognized by GRUB:

 # grub-probe /
 zfs

And that the ZFS modules for GRUB are installed:

 # ls /boot/grub/zfs*
 /boot/grub/zfs.mod  /boot/grub/zfsinfo.mod

Note that after Ubuntu 13, these are now in /boot/grub/i386/pc/zfs*

 # ls /boot/grub/i386-pc/zfs*
 /boot/grub/i386-pc/zfs.mod  /boot/grub/i386-pc/zfsinfo.mod

Otherwise, check the troubleshooting notes for GRUB below.

6.2 Refresh the initrd files:

 # update-initramfs -c -k all
 update-initramfs: Generating /boot/initrd.img-3.2.0-40-generic

6.3 Update the boot configuration file:

 # update-grub
 Generating grub.cfg ...
 Found linux image: /boot/vmlinuz-3.2.0-40-generic
 Found initrd image: /boot/initrd.img-3.2.0-40-generic
 done

Verify that boot=zfs appears in the boot configuration file:

 # grep boot=zfs /boot/grub/grub.cfg
 linux /ROOT/ubuntu-1/@/boot/vmlinuz-3.2.0-40-generic root=/dev/sda2 ro boot=zfs $bootfs quiet splash $vt_handoff
 linux /ROOT/ubuntu-1/@/boot/vmlinuz-3.2.0-40-generic root=/dev/sda2 ro single nomodeset boot=zfs $bootfs

6.4 Install the boot loader to the MBR like this:

 # grub-install $(readlink -f /dev/disk/by-id/scsi-SATA_disk1)
 Installation finished. No error reported.

Do not reboot the computer until you get exactly that result message. Note that you are installing the loader to the whole disk, not a partition.

Note: The readlink is required because recent GRUB releases do not dereference symlinks.

Step 7: Cleanup and First Reboot

7.1 Exit from the chroot environment back to the LiveCD environment:

 # exit

7.2 Run these commands in the LiveCD environment to dismount all filesystems:

 # umount /mnt/boot/grub
 # umount /mnt/dev
 # umount /mnt/proc
 # umount /mnt/sys
 # zfs umount -a
 # zpool export rpool

The zpool export command must succeed without being forced or the new system will fail to start.

7.3 We're done!

 # reboot

Before Step 6, I tried doing all of the steps above without modification. As expected, the system still wouldn’t work. (The thinking here was that this was a simple zpool.cache issue.)

On another attempt, I re-did the apt-get update and upgrade. I had an ‘aha’ moment and did:

# apt-get dist-upgrade

When I was patching for heartbleed, I was just doing an apt-get upgrade. The dist-upgrade will even install packages that are normally held back (like kernels). This minor mistake was probably what caused all of this. Anyways, after the dist-upgrade succeeded, I had some success. The system would boot and get to the point where the network was up and I could attempt to ssh in. However, I couldn’t get into my account.

At this point, my theory was that the ‘big’ pool wasn’t getting imported and mounted. That pool had my home directory. Any issue with it mounting would prevent me from logging in.

I thought I might have enough of a working system to fix the issue. I tried booting into recovery kernel via the ‘grub’ menu. When I chose the recovery kernel, I never got the proper grub recovery menu or to a shell. The system would boot and load the kernel + modules, but then it would hang. I then rebooted and edited the grub boot commands. I added a ‘1’ in order to get into single user mode. Same result, no shell. I edited the grub commands at boot again. I changed ro -> rw and adding init=/bin/bash. I then booted and got into a shell. I cannot remember if the ‘rpool’ was imported/mounted at this point. Either the rpool was mounted or I mounted it. I then noticed that my big pool wasn’t mounted, but I wasn’t sure if it would mount at this stage anyway.

I then imported my big pool. On my ‘big’ pool, I also have a zfs mount to /var/cache. Since the mount point wasn’t empty, it complained. I then cleared out the old /var/cache and remounted. All worked and looked good. However, like an idiot, I hit CTRL-D to exit the shell and the kernel immediately did a panic...(as expected). Doh! I now had the expectation that the system didn’t properly sync to disk and I would have to redo everything from the last boot.

I rebooted the system without editing the boot parameters, and the system looked like it was working. I tried to ssh in again… and it worked! wow! I guess just enough info did get synced for the system to resume.

I rebooted to verify that the system would come back up again. I also verified that I was completely up to date with apt-get update. I also verified that I had the patched OpenSSL. All of this was good. After my last reboot check, I did my normal zfs back ups to another system.

Now, why did it work? Definitely there was a self-inflicted issue with the dist-upgrade. Why didn’t the ‘big’ pool mount? Not sure. I didn’t redo the zpool.cache in the ram disks. I suspect it was just an issue of the ‘big’ pool getting imported and mounted correctly.

| improve this answer | |
  • 1
    Please if you write another answers in future, don't skip important information. Also try to keep it short if possible. – Anwar Dec 2 '16 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.