1

I have an external USB drive, and I run an hourly rsync to it as a backup. This has been working fine for years. This weekend, I got two new 2Tb internal drives, and decided it was time to re-install Ubuntu from scratch to clear out all the old cruft.

About once a day since the re-install, the backup script hangs hard, usually in the "rm -rf" I do before the rsync. By the time I notice the problem, my load average is in the stratosphere and climbing fast (one time, it was over 150), but anything that doesn't touch the drive seems to be running fine. One thing that I find suspicious is that something, I don't know what, is doing a "smartctl" and a "hdparm" command on the USB drive. I'm pretty sure smartctl isn't supposed to run on external drives. I can't figure out what's doing it, either. Here's part of ps auwwfx when it's hung:

root      7310  0.0  0.0   4248   352 ?        D    20:15   0:00 /sbin/hdparm -C /dev/sdd
root      7808  0.0  0.0  17372  1632 ?        D    20:15   0:00 /usr/sbin/smartctl -a -n standby -A -i /dev/sdd
(repeated every 5 minutes between the time the drive hung and now)

Why is this happening, and how can I stop it?

update It turns out that it's a munin plugin "hddtemp_smartctl" that's doing those two commands, but they work most of the day and then suddenly they start hanging. So I'm not closer to an answer.

update I also found this in kern.log the last time it happened:

Sep 21 23:18:01 allhats2 kernel: [52652.707110] INFO: task kjournald:4380 blocked for more than 120 seconds.
Sep 21 23:18:01 allhats2 kernel: [52652.707113] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 21 23:18:01 allhats2 kernel: [52652.707114] kjournald       D ffffffff81806200     0  4380      2 0x00000000
Sep 21 23:18:01 allhats2 kernel: [52652.707117]  ffff8803661e1c10 0000000000000046 ffff8803fd890148 ffff880404f17000
Sep 21 23:18:01 allhats2 kernel: [52652.707120]  ffff8803661e1fd8 ffff8803661e1fd8 ffff8803661e1fd8 00000000000137c0
Sep 21 23:18:01 allhats2 kernel: [52652.707122]  ffff880404e89700 ffff880361304500 ffff8803661e1bf0 ffff88041f5d4080
Sep 21 23:18:01 allhats2 kernel: [52652.707125] Call Trace:
Sep 21 23:18:01 allhats2 kernel: [52652.707130]  [<ffffffff811a8a40>] ? __wait_on_buffer+0x30/0x30
Sep 21 23:18:01 allhats2 kernel: [52652.707133]  [<ffffffff8165850f>] schedule+0x3f/0x60
Sep 21 23:18:01 allhats2 kernel: [52652.707135]  [<ffffffff816585bf>] io_schedule+0x8f/0xd0
Sep 21 23:18:01 allhats2 kernel: [52652.707137]  [<ffffffff811a8a4e>] sleep_on_buffer+0xe/0x20
Sep 21 23:18:01 allhats2 kernel: [52652.707139]  [<ffffffff81658ddf>] __wait_on_bit+0x5f/0x90
Sep 21 23:18:01 allhats2 kernel: [52652.707140]  [<ffffffff811a8a40>] ? __wait_on_buffer+0x30/0x30
Sep 21 23:18:01 allhats2 kernel: [52652.707142]  [<ffffffff81658e8c>] out_of_line_wait_on_bit+0x7c/0x90
Sep 21 23:18:01 allhats2 kernel: [52652.707145]  [<ffffffff8108ab20>] ? autoremove_wake_function+0x40/0x40
Sep 21 23:18:01 allhats2 kernel: [52652.707146]  [<ffffffff811a8a3e>] __wait_on_buffer+0x2e/0x30
Sep 21 23:18:01 allhats2 kernel: [52652.707149]  [<ffffffff81257534>] journal_commit_transaction+0x484/0xfc0
Sep 21 23:18:01 allhats2 kernel: [52652.707152]  [<ffffffff8125b5eb>] kjournald+0xeb/0x250
Sep 21 23:18:01 allhats2 kernel: [52652.707154]  [<ffffffff8108aae0>] ? add_wait_queue+0x60/0x60
Sep 21 23:18:01 allhats2 kernel: [52652.707155]  [<ffffffff8125b500>] ? commit_timeout+0x10/0x10
Sep 21 23:18:01 allhats2 kernel: [52652.707157]  [<ffffffff8108a03c>] kthread+0x8c/0xa0
Sep 21 23:18:01 allhats2 kernel: [52652.707160]  [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
Sep 21 23:18:01 allhats2 kernel: [52652.707162]  [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
Sep 21 23:18:01 allhats2 kernel: [52652.707163]  [<ffffffff81664b70>] ? gs_change+0x13/0x13
0

1 Answer 1

1

I did an hdparm -S 0 and it seems to have fixed the problem. At least I think so - it's been 24 hours with no freezeups.

update 48 hours later and it froze up again.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .