I have 3x 14TB toshiba drives mdadm-ed (/dev/md0) raid 5'd together that are setup as a bcache. I have a 256GB fast SSD as the front of the bache.

write-back is enabled on bcache.

After a few days, the device (/dev/bcache0) becomes extremely slow. I mean like 1000th of it's normal speed.

My 2 questions are:

  • For /dev/md0, what tuning should I do for those toshiba drives? It's 4k chunks 64k blocks.

  • Is there any bcache tuning I could do?

I'm not even really sure what other info I should put here. But if you ask, I'll update this post. Thanks!

Update 1- my IOSTAT while getting 100mb/sec read, only 3mb/sec write: https://pastebin.com/wKKf4LTq

The computer is an amd 2990wx w/32gb ram. CPU isn't the issue.

My old 3770k from 2010ish would get hard better read and write speeds than this. It's got to be some sort of setting or tuning. Thanks!

Update 2- While the system is running normally, below is the hdparm output. hdparm takes to long to run when it's not running normally.

/dev/md0:
 Timing cached reads:   11148 MB in  2.00 seconds = 5578.33 MB/sec
 Timing buffered disk reads: 1372 MB in  3.00 seconds = 456.84 MB/sec
/dev/bcache0:
 Timing cached reads:   12564 MB in  2.00 seconds = 6286.57 MB/sec
 Timing buffered disk reads: 1226 MB in  3.00 seconds = 408.66 MB/sec

Thanks!

  • I hope you understand, that what you have is a very unusual setup because the cache device is larger than the backing device. What exactly did you try to accomplish by setting up the bcache? Try dropping the bcache altogether and use the SSD directly. You will benefit from 10 times bigger storage AND much better performance. – Adam Ryczkowski Sep 12 at 16:07
  • @AdamRyczkowski oops! I fixed that typo. 14TB! Not GB... – Chemdream Sep 12 at 16:31
  • What exactly are the speeds of each of those three devices (i.e. SSD, HDD and MD0)? Do you mean sequential read or random read? Just write at least the first significant digit. – Adam Ryczkowski Sep 12 at 18:30
  • The SSD is a SAMSUNG 860 Pro. Max Sequential Read Up to 560 MBps Max Sequential Write Up to 530 MBps 4KB Random Read Random (QD1): Up to 11,000 IOPS Random (QD32): Up to 100,000 IOPS 4KB Random Write Random (QD1): Up to 43,000 IOPS Random (QD32): Up to 90,000 IOPS The 14TB's are Toshiba MG07ACA14TE. 248 MiB/s Typ. 242 MiB/s – Chemdream Sep 12 at 19:07
  • Ok, but that is only two devices - SSD i HDD. Could you also put the speed of the md0? I want to get a feeling of the degree of the slowdown you are experiencing. Also it is better to edit the question rather than posting the information in comments. Just to keep all the relevant info in one place. – Adam Ryczkowski Sep 13 at 5:35

With Samsung TLC memory, I'd stick to 512k bucket size. This will align with the page size for every 3 buckets (usually you would match the other way around but there's no sane way to align 1.5MB with any bucket size = 2^n). Use a sector size of 4k. BTW: This assumes, Samsung TLC uses 1.5MB page size but this is not officially documented somewhere. But 512k is still a safe value also for 2MB page size because it would align every 4 buckets.

Also, please align your data offset with your RAID5 setup. The bcache docs give some hints for that. It's very important to get that right. Personally, I didn't yet try such a setup but I guess [sysfs]/bdev*/partial_stripes_expensive may also be interesting in RAID-5.

I'm also guessing that the slowdowns show up when the cache has filled. You should disable discard for the cache, it's a synchronous operation for many drives due to firmware bugs. Instead, remove the bcache cdev, trim the whole partition, then resize the partition to 80-90% of its original size, align it to a 2MB boundary, and recreate bcache. Then, never touch this free partition space, it allows the drive to do background wear-leveling, discard is no longer needed then. You could create a protective partition to reserve this space, this makes it also easy to trim the reserved space.

To recreate the cache device, detach it from the backing device via sysfs, wait for completion, then unregister it, follow the steps for recreating it correctly, then attach the backing device back to the new cache. This can all be done online without reboot. But if you're not comfortable with it, make backups first.

  • 1
    I have empirically found that echo 0 > /sys/block/bcache0/bcache/writeback_percent forces the cache to be written to disk immediatly; otherwise, you will have to wait a long time for this to happen. Then revert it back to its orignal value (10) – solsTiCe Nov 21 at 15:56
  • Good point. This is probably due to the patches limiting writeback rate to favor better throughput for requests that bypass the writeback cache. But usually, it should automatically maximize writeback rate when flushing the cache on detach. That could be a bug... – hurikhan77 Nov 21 at 22:17

This had to still be building or re-indexing or something. Out of no where, it started running very fast.

So anyone else that has this issue, look at your mdadm status. If it's doing anything, that might be the cause. Also, by default, it reindexes the first Sunday of every month.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.