Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

On a pretty high traffic web site running in virtual containers (VMware) and lacking local storage, we've managed to increase throughput (requests per second) significantly by switching from logging directly into log-files (which reside on remote network storage) to rsyslogd.

Essentially we've switched from synchronous to asynchronous logging. The web server workers write using syslog(3) to some memory buffer and rsyslogd(8) sends the data to an actual file in parallel, and at its own pace, so processes don't block on IO when logging.

So far so good. The problem is that ocassionally rsyslogd is prevented from writing (e.g. a momentary/prolonged network outage) and the incoming buffer quickly fills up.

My questions are:

  • Can a client ever block when writing to rsyslogd using syslog(3)?
  • Is there a way to look at rsyslogd stats, e.g. how big/full is the buffer?
  • Is there a way to increase the size of the rsyslogd incoming buffer?
share|improve this question
    
Did you ever solve this? If so, I'd be interested in reading your answer. –  djeikyb Dec 31 '13 at 17:21
    
@djeikyb: sorry no. I see interest (votes on the question) but no one has answered it yet. Looks like this requires a source-code dive. –  arielf Jan 1 at 9:24
    
You don't say which web server you're using. Maybe you shouldn't be using syslog at all. Does Apache, for example, use syslog to log, or just write to log files? Logging into a database is another possibility. –  blujay May 19 at 3:23

2 Answers 2

The answer to your first question is:

Yes, any call to syslog() is blocking. Maybe for a very short time, but it's still a synchronous call involving a file descriptor. See man 3 syslog for more detials.

Unless your servers use asynchronous architectures and primitives, there will always be some locking. Thsi can be mitigated, but not eliminated, for example by using a separatethread for logging. For the other two questions I don't really know but inspection to the rsyslogd source code (as well as to the one for the syslog() family of functions) is the only way to know.

More in general, if you move the logging to an external server via the UDP:514 "network syslog protocol", then you bring the possibilities to create locks to almost zero. With the drawback of possible loss of some logging during high loads.

First, in the "originating" servers you need to ensure all logging happens via syslog. For example, in Apache2 you need to specify:

ErrorLog "syslog:daemon"

For other servers please refer to the proper man page. If you cannot ensure this, please keep in mind that logging on filesystems can create

Second, in the originating rsyslogd configuration you ask to direct all syslog traffic for the facility you choose ("daemon" in this example) to one or more external syslog servers. In the rsyslog configuration file you can specify:

daemon.* @192.168.128.1
daemon.* @192.168.254.1

to have two copies of the logs to be sent to two different servers at the same time.

Third, in the destination server(s) you enable the reception of the syslog message over UDP:514. It's in the (destination) rsyslogd configuration file and is normally disabled by defualt (it'd be enough to remove the leading #s:

$ModLoad imudp
$UDPServerRun 514

Fourth, optional but highly recommended, I would also enable high resolution timestamps:

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

Also this option is normally disabled by default (why on Earth?).

share|improve this answer

As far as I remember the default mode for the main message queue in rsyslog is fixed size array. It has a limit for 10k elements or so. Try to change this to linked list queue it should handle your occasional message bursts much better.

Yes, there are FixedArray and LinkedList queues.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.