On Ubuntu 18.04 both journald and rsyslog are installed. Both serve the same purpose of collecting log messages and storing them. So we have two programs doing the same work here.
What I could find out is that journald only saves log messages in its own journal files which can be queried with the
journalctl command. No readable files are written. Also, journal will always collect log output from all the services that were started by systemd, which should always be all of them because ultimately systemd starts all processes even from older schemes like init scripts.
Rsyslog also receives log messages, through some socket that works like syslog has worked the past 50 years or so. Applications have to send their messages to that socket and rsyslog will receive them. Not sure how journald actually receives anything. Then rsyslog forwards these received messages into different readable files as per its configuration.
In Ubuntu, both are installed but do not seem to be connected. At least rsyslog does not load the "imjournal" module with which it could "read" log messages from journald. I don't understand how rsyslog (actively) reads anything, I thought it only (passively) receives messages. But it's not using that anyway.
The file /var/log/syslog contains largely the same content as shown by
journalctl, just the output from
sudo is missing in the file. But I haven't compared it all.
So how does this work? Who sends log messages to where and who receives them and where are they forwarded to? If both destinations are filled, does every application have to send messages to both syslog and journald separately? If they are interconnected, why are no messages duplicated? Are they filtered out again? Which location is more complete and where should I look if I want the complete picture?
If any of this has changed in newer versions of Ubuntu, I'd also be interested in that, as I'm going to switch an older server to 20.04 next year.
Edit: Here are some more resources on the topic:
- Interaction of Rsyslog and Journal (Red Hat sysadmin guide)
- imjournal (rsyslog documentation)
- rsyslog journal integration, Video presentation by the rsyslog author
- The syslog system call
- Coexistence with systemd (rsyslog documentation)
All of this information leaves me a bit confused. It seems like journal receives all syslog data from the system call in the first place (my /dev/log points to /run/systemd/journal/dev-log), and rsyslog fetches it from journal through a socket that looks like the socket that is had used before there was journal (possibly /run/systemd/journal/syslog, implicitly used if it sees systemd around). Then rsyslog can do all its advanced log processing (including writing the legacy /var/log files). If I don't need any of that magic, I might just disable rsyslog. But I'm not sure about that at all!