I got things working using the excellent ZFS How-To (This how to is now replaced and newer How-Tos are recommended to be followed)
I booted an Ubuntu 12.04 live usb and used all the commands from step 1 of the HOWTO.
Step 1: Prepare The Install Environment
1.1 Start the Ubuntu LiveCD and open a terminal at the desktop.
1.2 Input these commands at the terminal prompt:
$ sudo -i
# apt-add-repository --yes ppa:zfs-native/stable
# apt-get update
# apt-get install debootstrap spl-dkms zfs-dkms ubuntu-zfs
1.3 Check that the ZFS filesystem is installed and available:
# modprobe zfs
# dmesg | grep ZFS:
ZFS: Loaded module v0.6.3-2~trusty, ZFS pool version 5000, ZFS filesystem version 5
Step 2: Disk Partitioning
This tutorial intentionally recommends MBR partitioning. GPT can be used instead, but beware of UEFI firmware bugs.
2.1 Run your favorite disk partitioner, like parted
or cfdisk
, on the primary storage device. /dev/disk/by-id/scsi-SATA_disk1
is the example device used in this document.
2.2 Create a small MBR primary partition of at least 8 megabytes. 256mb may be more realistic, unless space is tight. /dev/disk/by-id/scsi-SATA_disk1-part1
is the example boot partition used in this document.
2.3 On this first small partition, set type=BE
and enable the bootable
flag.
2.4 Create a large partition of at least 4 gigabytes. /dev/disk/by-id/scsi-SATA_disk1-part2
is the example system partition used in this document.
2.5 On this second large partition, set type=BF
and disable the bootable
flag.
The partition table should look like this:
# fdisk -l /dev/disk/by-id/scsi-SATA_disk1
Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sda1 * 1 1 8001 be Solaris boot
/dev/sda2 2 1305 10474380 bf Solaris
Remember: Substitute scsi-SATA_disk1-part1
and scsi-SATA_disk1-part2
appropriately below.
Hints:
* Are you doing this in a virtual machine? Is something in /dev/disk/by-id
missing? Go read the troubleshooting section.
* Recent GRUB releases assume that the /boot/grub/grubenv
file is writable by the stage2 module. Until GRUB gets a ZFS write enhancement, the GRUB modules should be installed to a separate filesystem in a separate partition that is grub-writable.
* If /boot/grub
is in the ZFS filesystem, then GRUB will fail to boot with this message: error: sparse file not allowed
. If you absolutely want only one filesystem, then remove the call to recordfail()
in each grub.cfg
menu stanza, and edit the /etc/grub.d/10_linux
file to make the change permanent.
* Alternatively, if /boot/grub
is in the ZFS filesystem you can comment each line with the text save_env
in the file /etc/grub.d/00_header
and run update-grub.
Step 3: Disk Formatting
3.1 Format the small boot partition created by Step 2.2 as a filesystem that has stage1 GRUB support like this:
# mke2fs -m 0 -L /boot/grub -j /dev/disk/by-id/scsi-SATA_disk1-part1
3.2 Create the root pool on the larger partition:
# zpool create -o ashift=9 rpool /dev/disk/by-id/scsi-SATA_disk1-part2
Always use the long /dev/disk/by-id/* aliases with ZFS. Using the /dev/sd* device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool.
Warning: The grub2-1.99 package currently published in the PPA for Precise does not reliably handle a 4k block size, which is ashift=12
.
Hints:
* # ls -la /dev/disk/by-id
will list the aliases.
* The root pool can be a mirror. For example, zpool create -o ashift=9 rpool mirror /dev/disk/by-id/scsi-SATA_disk1-part2 /dev/disk/by-id/scsi-SATA_disk2-part2
. Remember that the version and ashift matter for any pool that GRUB must read, and that these things are difficult to change after pool creation.
* If you are using a mirror with a separate boot partition as described above, don't forget to edit the grub.cfg file on the second HD partition so that the "root=" partition refers to that partition on the second HD also; otherwise, if you lose the first disk, you won't be able to boot from the second because the kernel will keep trying to mount the root partition from the first disk.
* The pool name is arbitrary. On systems that can automatically install to ZFS, the root pool is named "rpool" by default. Note that system recovery is easier if you choose a unique name instead of "rpool". Anything except "rpool" or "tank", like the hostname, would be a good choice.
* If you want to create a mirror but only have one disk available now you can create the mirror using a sparse file as the second member then immediately off-line it so the mirror is in degraded mode. Later you can add another drive to the spool and ZFS will automatically sync them. The sparse file won't take up more than a few KB so it can be bigger than your running system. Just make sure to off-line the sparse file before writing to the pool.
3.2.1 Create a sparse file at least as big as the larger partition on your HDD:
# truncate -s 11g /tmp/sparsefile
3.2.2 Instead of the command in section 3.2 use this to create the mirror:
# zpool create -o ashift=9 rpool mirror /dev/disk/by-id/scsi-SATA_disk1-part2 /tmp/sparsefile
3.2.3 Offline the sparse file. You can delete it after this if you want.
# zpool offline rpool /tmp/sparsefile
3.2.4 Verify that the pool was created and is now degraded.
# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
rpool 10.5G 188K 10.5G 0% 1.00x DEGRADED -
3.3 Create a "ROOT" filesystem in the root pool:
# zfs create rpool/ROOT
3.4 Create a descendant filesystem for the Ubuntu system:
# zfs create rpool/ROOT/ubuntu-1
On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through pkg image-update
or beadm
. Similar functionality for APT is possible but currently unimplemented.
3.5 Dismount all ZFS filesystems.
# zfs umount -a
3.6 Set the mountpoint
property on the root filesystem:
# zfs set mountpoint=/ rpool/ROOT/ubuntu-1
3.7 Set the bootfs
property on the root pool.
# zpool set bootfs=rpool/ROOT/ubuntu-1 rpool
The boot loader uses these two properties to find and start the operating system. These property names are not arbitrary.
Hint: Putting rpool=MyPool
or bootfs=MyPool/ROOT/system-1
on the kernel command line overrides the ZFS properties.
3.9 Export the pool:
# zpool export rpool
Don't skip this step. The system is put into an inconsistent state if this command fails or if you reboot at this point.
(I have include Step 2 and Step 3 for completeness, but I skipped them, since everything was already partitioned and formatted.)
I then ran zpool list
to verify that no pools were loaded. (None were)
I then did only Step 4.1
zpool import -d /dev/disk/by-id -R /mnt rpool
The full step 4 is below
Step 4: System Installation
Remember: Substitute "rpool" for the name chosen in Step 3.2.
4.1 Import the pool:
# zpool import -d /dev/disk/by-id -R /mnt rpool
If this fails with "cannot import 'rpool': no such pool available", you can try import the pool without the device name eg:
# zpool import -R /mnt rpool
4.2 Mount the small boot filesystem for GRUB that was created in step 3.1:
# mkdir -p /mnt/boot/grub
# mount /dev/disk/by-id/scsi-SATA_disk1-part1 /mnt/boot/grub
4.4 Install the minimal system:
# debootstrap trusty /mnt
The debootstrap
command leaves the new system in an unconfigured state. In Step 5, we will only do the minimum amount of configuration necessary to make the new system runnable.
I then checked ls -al /mnt
to make sure it looked ok.
I then skipped to Step 5.4. It was time to chroot
into the imported rpool.
# mount --bind /dev /mnt/dev
# mount --bind /proc /mnt/proc
# mount --bind /sys /mnt/sys
# chroot /mnt /bin/bash --login
At this point, we are now in the proper state to fix the system.
The complete step 5 was
Step 5: System Configuration
5.1 Copy these files from the LiveCD environment to the new system:
# cp /etc/hostname /mnt/etc/
# cp /etc/hosts /mnt/etc/
5.2 The /mnt/etc/fstab
file should be empty except for a comment. Add this line to the /mnt/etc/fstab
file:
/dev/disk/by-id/scsi-SATA_disk1-part1 /boot/grub auto defaults 0 1
The regular Ubuntu desktop installer may add dev
, proc
, sys
, or tmp
lines to the /etc/fstab
file, but such entries are redundant on a system that has a /lib/init/fstab
file. Add them now if you want them.
5.3 Edit the /mnt/etc/network/interfaces
file so that it contains something like this:
# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet dhcp
Customize this file if the new system is not a DHCP client on the LAN.
5.4 Make virtual filesystems in the LiveCD environment visible to the new system and chroot
into it:
# mount --bind /dev /mnt/dev
# mount --bind /proc /mnt/proc
# mount --bind /sys /mnt/sys
# chroot /mnt /bin/bash --login
5.5 Install PPA support in the chroot environment like this:
# locale-gen en_US.UTF-8
# apt-get update
# apt-get install ubuntu-minimal software-properties-common
Even if you prefer a non-English system language, always ensure that en_US.UTF-8 is available. The ubuntu-minimal package is required to use ZoL as packaged in the PPA.
5.6 Install ZFS in the chroot environment for the new system:
# apt-add-repository --yes ppa:zfs-native/stable
# apt-add-repository --yes ppa:zfs-native/grub
# apt-get update
# apt-get install --no-install-recommends linux-image-generic linux-headers-generic
# apt-get install ubuntu-zfs
# apt-get install grub2-common grub-pc
# apt-get install zfs-initramfs
# apt-get dist-upgrade
Warning: This is the second time that you must wait for the SPL and ZFS modules to compile. Do not try to skip this step by copying anything from the host environment into the chroot environment.
Note: This should install a kernel package and its headers, a patched mountall and dkms packages. Double-check that you are getting these packages from the PPA if you are deviating from these instructions in any way.
Choose /dev/disk/by-id/scsi-SATA_disk1
if prompted to install the MBR loader.
Ignore warnings that are caused by the chroot environment like:
Can not write log, openpty() failed (/dev/pts not mounted?)
df: Warning: cannot read table of mounted file systems
mtab is not present at /etc/mtab.
5.7 Set a root password on the new system:
# passwd root
Hint: If you want the ubuntu-desktop package, then install it after the first reboot. If you install it now, then it will start several process that must be manually stopped before dismount.
Step 6: GRUB Installation
Remember: All of Step 6 depends on Step 5.4 and must happen inside the chroot environment.
6.1 Verify that the ZFS root filesystem is recognized by GRUB:
# grub-probe /
zfs
And that the ZFS modules for GRUB are installed:
# ls /boot/grub/zfs*
/boot/grub/zfs.mod /boot/grub/zfsinfo.mod
Note that after Ubuntu 13, these are now in /boot/grub/i386/pc/zfs*
# ls /boot/grub/i386-pc/zfs*
/boot/grub/i386-pc/zfs.mod /boot/grub/i386-pc/zfsinfo.mod
Otherwise, check the troubleshooting notes for GRUB below.
6.2 Refresh the initrd files:
# update-initramfs -c -k all
update-initramfs: Generating /boot/initrd.img-3.2.0-40-generic
6.3 Update the boot configuration file:
# update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.2.0-40-generic
Found initrd image: /boot/initrd.img-3.2.0-40-generic
done
Verify that boot=zfs
appears in the boot configuration file:
# grep boot=zfs /boot/grub/grub.cfg
linux /ROOT/ubuntu-1/@/boot/vmlinuz-3.2.0-40-generic root=/dev/sda2 ro boot=zfs $bootfs quiet splash $vt_handoff
linux /ROOT/ubuntu-1/@/boot/vmlinuz-3.2.0-40-generic root=/dev/sda2 ro single nomodeset boot=zfs $bootfs
6.4 Install the boot loader to the MBR like this:
# grub-install $(readlink -f /dev/disk/by-id/scsi-SATA_disk1)
Installation finished. No error reported.
Do not reboot the computer until you get exactly that result message. Note that you are installing the loader to the whole disk, not a partition.
Note: The readlink is required because recent GRUB releases do not dereference symlinks.
Step 7: Cleanup and First Reboot
7.1 Exit from the chroot
environment back to the LiveCD environment:
# exit
7.2 Run these commands in the LiveCD environment to dismount all filesystems:
# umount /mnt/boot/grub
# umount /mnt/dev
# umount /mnt/proc
# umount /mnt/sys
# zfs umount -a
# zpool export rpool
The zpool export
command must succeed without being forced or the new system will fail to start.
7.3 We're done!
# reboot
Before Step 6, I tried doing all of the steps above without modification. As expected, the system still wouldn’t work. (The thinking here was that this was a simple zpool.cache issue.)
On another attempt, I re-did the apt-get update and upgrade. I had an ‘aha’ moment and did:
# apt-get dist-upgrade
When I was patching for heartbleed, I was just doing an apt-get upgrade
. The dist-upgrade will even install packages that are normally held back (like kernels). This minor mistake was probably what caused all of this. Anyways, after the dist-upgrade succeeded, I had some success. The system would boot and get to the point where the network was up and I could attempt to ssh in. However, I couldn’t get into my account.
At this point, my theory was that the ‘big’ pool wasn’t getting imported and mounted. That pool had my home directory. Any issue with it mounting would prevent me from logging in.
I thought I might have enough of a working system to fix the issue. I tried booting into recovery kernel via the ‘grub’ menu. When I chose the recovery kernel, I never got the proper grub recovery menu or to a shell. The system would boot and load the kernel + modules, but then it would hang. I then rebooted and edited the grub boot commands. I added a ‘1’ in order to get into single user mode. Same result, no shell. I edited the grub commands at boot again. I changed ro
-> rw
and adding init=/bin/bash
. I then booted and got into a shell. I cannot remember if the ‘rpool’ was imported/mounted at this point. Either the rpool was mounted or I mounted it. I then noticed that my big pool wasn’t mounted, but I wasn’t sure if it would mount at this stage anyway.
I then imported my big pool. On my ‘big’ pool, I also have a zfs mount to /var/cache
. Since the mount point wasn’t empty, it complained. I then cleared out the old /var/cache
and remounted. All worked and looked good. However, like an idiot, I hit CTRL-D to exit the shell and the kernel immediately did a panic...(as expected). Doh! I now had the expectation that the system didn’t properly sync to disk and I would have to redo everything from the last boot.
I rebooted the system without editing the boot parameters, and the system looked like it was working. I tried to ssh in again… and it worked! wow! I guess just enough info did get synced for the system to resume.
I rebooted to verify that the system would come back up again. I also verified that I was completely up to date with apt-get update. I also verified that I had the patched OpenSSL. All of this was good. After my last reboot check, I did my normal zfs back ups to another system.
Now, why did it work? Definitely there was a self-inflicted issue with the dist-upgrade. Why didn’t the ‘big’ pool mount? Not sure. I didn’t redo the zpool.cache in the ram disks. I suspect it was just an issue of the ‘big’ pool getting imported and mounted correctly.