Ubuntu 14.04 - logrotate does not rotate /var/log/* (rsyslog config)

Question

I have a bunch of Ubuntu 14.04 servers I manage where logrotate is not rotating anything under /var/log/*. I'm going to assume its an issue with the /etc/logrotate.d/rsyslog as I see other entries in /etc/logrotate.d/ rotating logs properly.

These servers are deployed using CHEF. However I don't see anything in the recipes, etc. that are referencing any functional changes to logrotate. There are a few scripts added to /etc/logrotate.d and those scripts do run every day as expected. Is it possible that one script under /etc/logrotate.d could affect another (I wouldn't think so)? Similar servers using the same CHEF recipes are configured on Ubuntu 12.04 boxes do not have this symptom of logs not rotating under /var/log/*.

Running /usr/sbin/logrotate -d /etc/logrotate.conf, I get the following (omitting anything not rsyslog related):

rotating pattern: /var/log/syslog
 after 1 days (7 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 104
considering log /var/log/syslog
  log does not need rotating
switching euid to 0 and egid to 0

rotating pattern: /var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
 weekly (4 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 104
considering log /var/log/mail.info
  log does not need rotating
considering log /var/log/mail.warn
  log /var/log/mail.warn does not exist -- skipping
considering log /var/log/mail.err
  log /var/log/mail.err does not exist -- skipping
considering log /var/log/mail.log
  log does not need rotating
considering log /var/log/daemon.log
  log does not need rotating
considering log /var/log/kern.log
  log does not need rotating
considering log /var/log/auth.log
  log does not need rotating
considering log /var/log/user.log
  log does not need rotating
considering log /var/log/lpr.log
  log /var/log/lpr.log does not exist -- skipping
considering log /var/log/cron.log
  log /var/log/cron.log does not exist -- skipping
considering log /var/log/debug
  log does not need rotating
considering log /var/log/messages
  log does not need rotating
not running postrotate script, since no logs were rotated
switching euid to 0 and egid to 0

When I look in /var/log, I can see that nothing is being rotated:

$ ls -l /var/log
total 34116
-rw-r--r-- 1 root      root    19512 Jan  9 07:15 alternatives.log
drwxr-xr-x 2 root      root     4096 Jan  7 20:28 apt
-rw-r----- 1 syslog    adm   3725622 Jan 12 19:50 auth.log
-rw-r--r-- 1 root      root     2481 Jan 10 03:46 boot.log
-rw-rw---- 1 root      utmp        0 Apr 11  2013 btmp
-rw-r--r-- 1 syslog    adm      6170 Jan  7 20:11 cloud-init.log
drwxr-xr-x 2 root      root     4096 Nov 18  2011 cron-apt
-rw-r----- 1 root      adm     19724 Jan 12 19:21 daemon.log
-rw-r----- 1 root      adm      5944 Jan 10 03:46 debug
drwxr-xr-x 2 root      root     4096 Oct 10  2012 dist-upgrade
-rw-r--r-- 1 root      adm     15312 Jan 10 03:46 dmesg
-rw-r--r-- 1 root      adm     15312 Jan  8 04:18 dmesg.0
-rw-r--r-- 1 root      adm      5451 Jan  7 20:11 dmesg.1.gz
-rw-r--r-- 1 root      root       28 Jan  7 20:11 dmesg.2.gz
-rw-r--r-- 1 root      root   162648 Jan 12 07:10 dpkg.log
drwxr-xr-x 2 root      root     4096 Apr 11  2013 fsck
-rw-r----- 1 syslog    adm     81900 Jan 10 03:46 kern.log
drwxr-xr-x 2 landscape root     4096 Jan  7 20:11 landscape
-rw-rw-r-- 1 root      utmp   292584 Jan 12 19:20 lastlog
drwxr-xr-x 2 root      root     4096 Jan  7 20:43 logstash
-rw-r----- 1 syslog    adm         0 Jan  7 20:11 mail.err
-rw-r----- 1 root      adm   9433681 Jan 12 19:50 mail.info
-rw-r----- 1 syslog    adm   9433797 Jan 12 19:50 mail.log
-rw-r----- 1 root      adm         0 Jan  7 20:43 mail.warn
-rw-r----- 1 root      adm     92617 Jan 12 07:10 messages
-rw-r----- 1 root      adm       519 Jan  7 20:43 monit.log
drwxr-s--- 2 mysql     adm      4096 Jan  7 20:45 mysql
-rw-r----- 1 mysql     adm         0 Jan  7 20:45 mysql.err
-rw-r----- 1 mysql     adm         0 Jan  7 20:45 mysql.log
drwxr-xr-x 2 root      root     4096 Jan  7 20:11 news
drwxr-xr-x 2 www-data  root     4096 Jan  7 20:41 nginx
-rw-r----- 1 syslog    adm  11460381 Jan 12 19:50 syslog
drwxr-xr-x 3 root      root     4096 Jan 12 19:45 sysstat
-rw-r--r-- 1 root      root    84672 Jan 10 03:46 udev
-rw-r----- 1 syslog    adm         0 Jan  7 20:11 ufw.log
drwxr-xr-x 2 root      root     4096 Jan  8 04:20 upstart
-rw-r----- 1 root      adm     41714 Jan 12 07:10 user.log
-rw-rw-r-- 1 root      utmp   206208 Jan 12 19:20 wtmp

Here is my version of logrotate (Stock Ubuntu 14.04):

$logrotate
logrotate 3.7.8 - Copyright (C) 1995-2001 Red Hat, Inc.
This may be freely redistributed under the terms of the GNU Public License

Usage: logrotate [-dfv?] [-d|--debug] [-f|--force] [-m|--mail=command] [-s|--state=statefile] [-v|--verbose] [-?|--help]
        [--usage] [OPTION...] <configfile>

Logrotate is in cron.daily (default config):

$ ls /etc/cron.daily/logrotate
/etc/cron.daily/logrotate

The default crontab config was never changed:

$ cat /etc/crontab
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user  command
17 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
25 6    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6    * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6    1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

The /etc/logrotate.conf file:

$ cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly

# use the syslog group by default, since this is the owning group
# of /var/log/syslog.
su root syslog

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}

# system-specific logs may be configured here

The /etc/logrotate.d/rsyslog file:

$ cat rsyslog
/var/log/syslog
{
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        reload rsyslog >/dev/null 2>&1 || true
    endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
    rotate 4
    weekly
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        reload rsyslog >/dev/null 2>&1 || true
    endscript
}

Answer 1

Same issue happened and resolved on Ubuntu 14.04 server with below change in /etc/logrotate.d/rsyslog

from:

/var/log/syslog
{
    rotate 7
    daily
    ...
    postrotate
        reload rsyslog >/dev/null 2>&1 || true
    endscript
}

to this:

{
    rotate 7
    daily
    ...
    postrotate
        service rsyslog rotate >/dev/null 2>&1 || true
    endscript
}

for each rotated rsyslog files. I triggered logrotate manually with "/usr/sbin/logrotate /etc/logrotate.conf" and tested it with logger.

Answer 2

I found that removing the non-existent files from the file pattern solved the problem. With the non-existent files configured, although the existing files were considered on the right interval, the part of the rotation that moves the original never happened. Clearly a bug I suggest.