Plymouth check gets stuck, preventing service operation

Question

This is a strange one. If I do, say

service jenkins start

it will always freeze/get stuck, like waiting for some input. Same is with all services and start/stop commands. However, if I do it like

service jenkins start | cat

it will work without issues. Also piping to /dev/null works. Using set -x, I've managed to track down the freeze at

+ '[' -z ']'
+ FANCYTTY=1
+ case "$FANCYTTY" in
+ true
+ /usr/bin/tput xenl
++ /usr/bin/tput cols
+ COLS=144
+ '[' 144 ']'
+ '[' 144 -gt 6 ']'
++ /usr/bin/expr 144 - 7
+ COL=137
+ log_use_plymouth
+ '[' n = y ']'
+ plymouth --ping

And indeed, it is always plymouth --ping that freezes. It is reproducible directly from shell as well. Any ideas to troubleshoot this?

Ubuntu version is 14.04.01. Plymouth is supposed to log to /var/log/boot.log, but I don't see anything useful there, just "Stopping" and "Starting" lines that have OK as result.

Process list regards to plymouth looks like this:

$ ps -ef | grep plymou
root        478      1  0 Dec01 ?        00:00:04 plymouth-upstart-bridge
root        492      1  0 Dec01 ?        00:00:00 @sbin/plymouthd --mode=boot --attach-to-session
root       1710   1699  0 Dec01 ?        00:00:00 plymouth quit
root       1725   1411  0 Dec01 ?        00:00:00 plymouth --ping
root      48897   2140  0 12:12 pts/0    00:00:00 grep --color=auto plymou

strace plymouth --ping shows this:

connect(6, {sa_family=AF_LOCAL, sun_path=@"/org/freedesktop/plymouthd"}, 29) = 0
epoll_ctl(3, EPOLL_CTL_ADD, 6, {EPOLLERR|EPOLLHUP, {u32=31321216, u64=31321216}}) = 0
epoll_ctl(3, EPOLL_CTL_MOD, 6, {EPOLLERR|EPOLLHUP, {u32=31321216, u64=31321216}}) = 0
epoll_ctl(3, EPOLL_CTL_MOD, 6, {EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=31321216, u64=31321216}}) = 0
epoll_wait(3, {{EPOLLOUT, {u32=31321216, u64=31321216}}}, 64, -1) = 1
sendto(6, "P\0", 2, MSG_NOSIGNAL, NULL, 0) = 2
epoll_ctl(3, EPOLL_CTL_MOD, 6, {EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=31321216, u64=31321216}}) = 0
epoll_ctl(3, EPOLL_CTL_MOD, 6, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=31321216, u64=31321216}}) = 0
epoll_wait(3,

/var/log/upstart/plymouth.log has one line:

error: unexpectedly disconnected from boot status daemon

Answer 1

Turns out, when I was debugging by removing splash and quiet and replacing them with plymouth:debug in /etc/default/grub as adviced in Plymouth Debugging, all my issues seem to be gone. No need to edit logging scripts like above.

It might have been some race condition with plymouth on ubuntu startup which causes plymouth to get stuck. By avoiding splash I am able to avoid the problem.

Before I found that out, I was hacking by editing /lib/lsb/init.functions.d/50-ubuntu-logging to always return 0 for plymouth usage (first function). That didn't help with restart/boot issues though.