On a pretty high traffic web site running in virtual containers (VMware) and lacking local storage, we've managed to increase throughput (requests per second) significantly by switching from logging directly into log-files (which reside on remote network storage) to rsyslogd.
Essentially we've switched from synchronous to asynchronous logging. The web server workers write using syslog(3) to some memory buffer and rsyslogd(8) sends the data to an actual file in parallel, and at its own pace, so processes don't block on IO when logging.
So far so good. The problem is that ocassionally rsyslogd is prevented from writing (e.g. a momentary/prolonged network outage) and the incoming buffer quickly fills up.
My questions are:
- Can a client ever block when writing to rsyslogd using syslog(3)?
- Is there a way to look at rsyslogd stats, e.g. how big/full is the buffer?
- Is there a way to increase the size of the rsyslogd incoming buffer?