Ceph drive failure and replacement procedure

Question

In my OpenStack deployment I'm using Ceph for block and object storage, which is reporting a healthy status. However, one of the hard drives is displaying a SMART error. I don't remember the exact error, but something to the effect of imminent failure.

What is the process for replacing the drive? Should I let it ride and let Ceph take care of it when it fails? Or should I replace the drive before Ceph notices anything?

I assume it's best to replace the drive before any damage is done, but in the case of using Ceph, I believe it can handle the failure. I'm not sure if in the JUJU Ceph charm has some magic that will make the replacement process easier one way or another.

I don't see any degradation in performance.

Answer 1

Ceph is designed to remain highly available despite individual failures of hard disks, nodes, or entire racks when a cluster is correctly designed. This is the primary reason for the 3-replica strategy on the data (RAID is not required nor is recommended).

You do not need to replace a hard disk before it fails, the standard replacement procedure would be to swap all failed hardware on a scheduled day of the month as part of your operational routine. You of course need to have enough free space in the cluster to accomodate your nominal hardware failure rate, but you need to have free space for other reasons in any enterprise storage system, so that goes without saying.

Once the drive fails, Ceph will create new copies of the data that was stored in the corresponding OSD elsewhere in the cluster, and restore the level of resiliency for that data to three replicas. That data is "degraded" to two replicas until this completes, but remains highly available as you still have at least two copies of it in the healthy part of the cluster.