This isn't a very big deal but it really irritates me somehow.
We have a large number of machines running Ubuntu 12.04 (server/amd64) which i frequently ssh into. We have the standard set-up for this version of Ubuntu, where there are a handful of scripts in /etc/update-motd.d that are used to build the MOTD, and /etc/pam.d/sshd is configured to print the MOTD on ssh log-in via pam_motd.so. Our MOTD scripts are all custom (we don't have any of the built-in Ubuntu ones on these machines), but everything else is absolutely stock Ubuntu server config. For the most part, this works fine.
However, once in a while, the MOTD is displayed only partially (or sometimes not at all). There is no error message either at the terminal or in any logs that i can find, it just doesn't display the complete output. If i cat /var/run/motd immediately after this occurs, that file does show the full (and up-to-date) output, though.
I know that run-parts has an option to stop processing scripts if one of them exits with a >0 status, but i don't think that's applicable in this case — often the output is truncated in the middle of the script execution. For example, one script might print out some lines like this:
mycompany header
hostname uname
cpu
ram
uptime
But the MOTD will stop immediately after mycompany header. These are very simple bash scripts and we do not have the -e option enabled in them, so i don't know how this can even happen. At first i thought that maybe there was a limit on the amount of time that PAM would wait for the MOTD to be generated, and that maybe sometimes (depending on load or whatever) we might be hitting that limit — but that doesn't seem to be the case. The MOTD is printed almost immediately every time, whether it's truncated or not.
The only other thing i can think of is that maybe run-parts is writing the dynamic MOTD file out one line at a time, and there is some kind of race condition where pam_motd is reading an incomplete version of that file.
However, i can't confirm that, because i have no idea what the mechanism is behind this functionality. My understanding is that PAM somehow fires off run-parts to update the file on log-in, but i can't find any references to run-parts or /etc/update-motd.d in the PAM source. All the pam_motd code seems to do is just read in the MOTD file.
The only possibly useful clue i could find is this: If i do watch -n 0.1 'ps aux | grep "run-parts"' in one terminal window, as i disconnect and re-connect to the same host in another window, i can see that, when the output is complete, there are always several processes that appear very briefly in the process list — but when it's truncated, there are only one or two processes (sometimes none at all).
edit: I should also add that i can't replicate this problem at all when i execute run-parts --lsbsysinit /etc/update-motd.d manually; it only seems to happen at ssh log-in.
How can i troubleshoot this further? How does PAM instruct the system to update the MOTD? Does it wait for that process to finish, or does it run concurrently? Is there a way to change that behaviour? Is there a debug option i can set somewhere to at least see what's happening? Is there any expected reason that run-parts might abort in the middle of a script?
cheers
