ZFS autoreplace unavailable drive

Question

I have the fallowing ZFS test pool setup in my vm:

    NAME        STATE     READ WRITE CKSUM
    pool        DEGRADED     0     0     0
      mirror-0  DEGRADED     0     0     0
        sdb     ONLINE       0     0     0
        sdc     ONLINE       0     0     0
    spares
      sdd       AVAIL

I a trying to test the ability to replace a drive that becomes unavailable automatically in a zfs pool. For example if for some reason the drive is removed from the system the spare dive could take its place immediately upon the drives removal. I have configured a spare drive and from the reading I have done, the way I understand it is that this should accomplish my goal. I have also set the autoreplace flag to on to tell zfs to replace any bad drive if detected. When remove a drive from the vm, say sdb and recheck the status of zfs it looks like the fallowing:

    NAME        STATE     READ WRITE CKSUM
    pool        DEGRADED     0     0     0
      mirror-0  DEGRADED     0     0     0
        sdb     UNAVAIL      0     0     0
        sdc     ONLINE       0     0     0
    spares
      sdd       AVAIL

The sdb device becomes unavailable but is never replaced by the spare. Is there a way i can get my use-case to work?

Answer 1

Intro

First of all for anyone who is having the same issue on Ubuntu 16.04, it is currently an ongoing bug and as of now, to my knowledge, has not been fixed. You can visit the github conversation here to see the bug I am referencing. Secondly I am writing this post as an enthusiast and intermediate linux user, I am not a developer or currently work on linux as a profession. That being said I will provide the info I have found on this issue to the best of my knowledge.

The Issue:

The hot spare functionality built into the zfsonlinux package currently does not work as stated in the man page. zpool man page:

Hot Spares
   ZFS allows devices to be associated with pools as "hot spares".
   These devices are not actively used in the pool, but when an active
   device fails, it is automatically replaced by a hot spare.

Note: Cold spare (a spare not replaced automatically) functionality is working but only in configurations consisting of mirrors or any of the raidz's. If you have a configuration like the fallowing:

NAME      STATE     READ WRITE CKSUM
pool      ONLINE       0     0     0
  sdb     ONLINE       0     0     0
  sdc     ONLINE       0     0     0
spares
  sdd     AVAIL

Upon the failure of a drive you will be unable to replace it with the spare due to the pool I/O is currently suspended error.

Attempted Solutions:

This is a list of attempted solutions I have gone through and the results is got:

1. ZFS auto replace: It turns out that according to the zpool man page and my understanding of it, the autoreplace functionality is not to replace a damaged drive with the spare but rather to replace the damaged drive itself with any new device found at its location. zpool man page:

   If set to "on", any new device, found in the same physical location
   as a device that previously belonged to the pool, is automatically 
   formatted and replaced.
   

   autoreplace is good to have on, but to achieve the functionality of a hot spare I don't believe it is needed. Im not able to confirm this other than through the man page because i was unable to get the spare to work.

2. ZED.rc: I later found from within the conversation on github that the ZED_SPARE_ON_CHECKSUM_ERRORS and ZED_SPARE_ON_IO_ERRORS flags needed to be set within the zed.rc. As @user121391 said in his answer to this post, the zed service/daemon is what determines the status of the pool and what is required to be done based on that status. You can see zed.d at work by running zpool events. After setting the flags, rebooting and, reconfiguring a new pool just in case, my testing of the hot spare came up negative again.

3. ZED.d scripts: In @Michael Kjörling's comment he mentioned some scripts packaged with ZED that are written to help with the hot spare scenario. I did indeed find the scripts located in /etc/zfs/zed.d. You can view these scripts on the github page here. From my understanding of zed.d, when an event is triggered the ZED daemon runs one of these scripts based on its status. That being said I did try to run the io-spare.sh and checksum-spare.sh manually and judging by the time they took to execute as well as the unchanged status of the pool, they didn't seem to be running correctly. Maybe this is a potential source to the issue.

My Plans

For Ubuntu 16.04 users of ZFS, to my knowledge there currently seems to be no fix to this issue. I plan to continue to use ZFS with the configuration of a mirrored or raidz(RAID5) depending the the performance i get in testing.(mirrored will be faster but I want to see the speed of raidz) A hot spare would increase the redundancy of both these configs but unfortunately we will have to wait until further development of the zfsonlinux project. v