SSH login wakes spun-down storage drives

Question

I'm running 12.04 on a home server/HTPC, and have several storage drives which spin down using values set in /etc/hdparm.conf.

Sometimes, logging in via ssh causes some or all of the drives to wake up. I set the udisks rule recommended in this post, and the spinning-up went from all the time to just some of the time.

I set audit rules, auditctl -w /dev/sde -p rwa -k e, for instance, to try to see what was waking them. Checking the audit results with ausearch, I see entries like this when I ssh in:

type=PATH msg=audit(04/16/2013 18:16:40.109:502) : item=0 name=/dev/sde inode=3010 dev=00:05 mode=block,660 ouid=root ogid=disk rdev=08:40 
type=CWD msg=audit(04/16/2013 18:16:40.109:502) :  cwd=/home/absqua 
type=SYSCALL msg=audit(04/16/2013 18:16:40.109:502) : arch=x86_64 syscall=open success=yes exit=3 a0=7fffcd697998 a1=800 a2=0 a3=0 items=1 ppid=14054 pid=14055 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=408 comm=hdparm exe=/sbin/hdparm key=e

So, my drives are being woken up by hdparm itself? How can I figure out what's calling it?

Answer 1

I was also struggling with this exact issue for a while, having tried audits, the mentioned udisks rule etc. But finally it was blktrace and blkparse that gave me the final hint of the problem: my sleeping disks were accessed by dumpe2fs on ssh login.

Apparently, the following script was the offender, scanning all drives to display which drives will be checked for errors at the next reboot on your motd at login.

/usr/lib/update-notifier/update-motd-fsck-at-reboot

The script only executes once every hour max, so that explains why it wasn't happening consistently. See lines 21-24:

if [ $(($stampt + 3600)) -lt $now ] || [ $stampt -gt $now ]; then
    #echo $stampt $now need update
        NEEDS_FSCK_CHECK=yes
fi

I decided to disable the automatic check on my system by replacing the variable assignation to : (do nothing). You could also increase the minimum time difference of 3600 seconds to something like a day or a week. But I found it preferable to run the script with --force during midnight maintenance when all my disks are awake anyhow. Since this script is purely informational it should be safe to just replace it with return 0 if you're not interested in this information.

I hope this helps anyone having the same or a similar issue. Even if it's another culprit on your system, I found blktrace and blkparse invaluable for diagnosing the problem.

Answer 2

Building on the answer by jvanhie: Another way to stop the /usr/lib/update-notifier/update-motd-fsck-at-reboot script from running every hour and during login, without modifying it, is to put this line in /etc/crontab:

01 * * * * root f=/var/lib/update-notifier/fsck-at-reboot ; test -e $f && touch $f

That will update the time stamp file checked by the script every hour if it exists, which will fool the script into thinking it has already run that hour.

You don't have to modify the script, and so you don't have to remember to carry forward any modifications after any system updates (though a more robust solution would issue an error warning if the time stamp file were not being used).