4

I have a (formerly working) logrotate setup for OpenResty that I took from a previous Ubuntu 18.04 installation. Yet the logrotate.service fails with this error now ...

error: error renaming /usr/local/openresty/nginx/logs/access.log.60.zst
to /usr/local/openresty/nginx/logs/access.log.61.zst: Read-only file system

... and I have difficulties understanding why. The new machines are running Ubuntu 20.04, but I don't see why that should make a difference in this case.

First, here is the config:

$ cat /etc/logrotate.d/custom-openresty

/usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
{
  daily
  rotate 60
  maxsize 1G
  missingok
  notifempty
  compress
  compresscmd /usr/bin/zstd
  uncompresscmd /usr/bin/unzstd
  compressoptions -9 --long -T1
  compressext .zst
  delaycompress
  sharedscripts
  postrotate
    test ! -f /usr/local/openresty/nginx/logs/nginx.pid || kill -USR1 `cat /usr/local/openresty/nginx/logs/nginx.pid`
 endscript
}

/etc/logrotate.conf is unchanged and looks like this:


# see "man logrotate" for details
# rotate log files weekly
weekly

# use the adm group by default, since this is the owning group
# of /var/log/syslog.
su root adm

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
#dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may be also be configured here.

Here is the state of my files (I would expect to get access.log.1 after logrotate ran):

$ ls -alhg /usr/local/openresty/nginx/logs/
total 10G
drwxr-xr-x  2 root 4.0K Sep 16 20:04 .
drwxr-xr-x 18 root 4.0K Sep 16 19:42 ..
-rw-r--r--  1 root  10G Sep 16 19:56 access.log
-rw-r--r--  1 root 5.5K Sep 16 19:56 error.log

However, the logrotate.service fails with this error:

~$ systemctl status logrotate.service 
● logrotate.service - Rotate log files
     Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2020-09-16 20:11:32 UTC; 4min 32s ago
TriggeredBy: ● logrotate.timer
       Docs: man:logrotate(8)
             man:logrotate.conf(5)
    Process: 27403 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
   Main PID: 27403 (code=exited, status=1/FAILURE)

$ sudo journalctl --unit logrotate.service

Sep 16 20:11:32 fetcher-scheduler systemd[1]: Starting Rotate log files...
Sep 16 20:11:32 fetcher-scheduler logrotate[27403]: error: error renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst: Read-only file system
Sep 16 20:11:32 fetcher-scheduler systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Sep 16 20:11:32 fetcher-scheduler systemd[1]: logrotate.service: Failed with result 'exit-code'.
Sep 16 20:11:32 fetcher-scheduler systemd[1]: Failed to start Rotate log files.

When I run it (without systemd) in debug mode as root, I get the following output:

# logrotate -v -d /etc/logrotate.d/custom-openresty

reading config file /etc/logrotate.d/custom-openresty
compress_prog is now /usr/bin/zstd
uncompress_prog is now /usr/bin/unzstd
compress_options is now  -9 --long -T1
compress_ext is now .zst
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
...
Creating new state

Handling 1 logs

rotating pattern: /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
 after 1 days (60 rotations)
empty log files are not rotated, log files >= 1073741824 are rotated earlier, old logs are removed
considering log /usr/local/openresty/nginx/logs/access.log
  Now: 2020-09-16 20:24
  Last rotated at 2020-09-16 20:11
  log needs rotating
considering log /usr/local/openresty/nginx/logs/error.log
  Now: 2020-09-16 20:24
  Last rotated at 2020-09-16 19:00
  log does not need rotating (log has been rotated at 2020-9-16 19:0, that is not day ago yet)
rotating log /usr/local/openresty/nginx/logs/access.log, log->rotateCount is 60
dateext suffix '-20200916'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /usr/local/openresty/nginx/logs/access.log.1 does not exist
renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst (rotatecount 60, logstart 1, i 60), 
renaming /usr/local/openresty/nginx/logs/access.log.59.zst to /usr/local/openresty/nginx/logs/access.log.60.zst (rotatecount 60, logstart 1, i 59), 
...
/logs/access.log.3.zst (rotatecount 60, logstart 1, i 2), 
renaming /usr/local/openresty/nginx/logs/access.log.1.zst to /usr/local/openresty/nginx/logs/access.log.2.zst (rotatecount 60, logstart 1, i 1), 
renaming /usr/local/openresty/nginx/logs/access.log.0.zst to /usr/local/openresty/nginx/logs/access.log.1.zst (rotatecount 60, logstart 1, i 0), 
log /usr/local/openresty/nginx/logs/access.log.61.zst doesn't exist -- won't try to dispose of it
renaming /usr/local/openresty/nginx/logs/access.log to /usr/local/openresty/nginx/logs/access.log.1
running postrotate script
running script with arg /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
: "
    test ! -f /usr/local/openresty/nginx/logs/nginx.pid || kill -USR1 `cat /usr/local/openresty/nginx/logs/nginx.pid`
"

To me, everything looks normal. If I run it, it also works:

# logrotate -v /etc/logrotate.d/custom-openresty
reading config file /etc/logrotate.d/custom-openresty
compress_prog is now /usr/bin/zstd
uncompress_prog is now /usr/bin/unzstd
compress_options is now  -9 --long -T1
compress_ext is now .zst
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state
...
Creating new state

Handling 1 logs

rotating pattern: /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
 after 1 days (60 rotations)
empty log files are not rotated, log files >= 1073741824 are rotated earlier, old logs are removed
considering log /usr/local/openresty/nginx/logs/access.log
  Now: 2020-09-16 20:26
  Last rotated at 2020-09-16 20:11
  log needs rotating
considering log /usr/local/openresty/nginx/logs/error.log
  Now: 2020-09-16 20:26
  Last rotated at 2020-09-16 19:00
  log does not need rotating (log has been rotated at 2020-9-16 19:0, that is not day ago yet)
rotating log /usr/local/openresty/nginx/logs/access.log, log->rotateCount is 60
dateext suffix '-20200916'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /usr/local/openresty/nginx/logs/access.log.1 does not exist
renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst (rotatecount 60, logstart 1, i 60), 
old log /usr/local/openresty/nginx/logs/access.log.60.zst does not exist
...
old log /usr/local/openresty/nginx/logs/access.log.2.zst does not exist
renaming /usr/local/openresty/nginx/logs/access.log.1.zst to /usr/local/openresty/nginx/logs/access.log.2.zst (rotatecount 60, logstart 1, i 1), 
old log /usr/local/openresty/nginx/logs/access.log.1.zst does not exist
renaming /usr/local/openresty/nginx/logs/access.log.0.zst to /usr/local/openresty/nginx/logs/access.log.1.zst (rotatecount 60, logstart 1, i 0), 
old log /usr/local/openresty/nginx/logs/access.log.0.zst does not exist
log /usr/local/openresty/nginx/logs/access.log.61.zst doesn't exist -- won't try to dispose of it
renaming /usr/local/openresty/nginx/logs/access.log to /usr/local/openresty/nginx/logs/access.log.1
running postrotate script

No error and in the end access.log.1 is created as expected:

# ls -algh /usr/local/openresty/nginx/logs/
total 10G
drwxr-xr-x  2 root 4.0K Sep 16 20:26 .
drwxr-xr-x 18 root 4.0K Sep 16 19:42 ..
-rw-r--r--  1 root  10G Sep 16 19:56 access.log.1
-rw-r--r--  1 root 5.5K Sep 16 19:56 error.log

Note that it said ...

log /usr/local/openresty/nginx/logs/access.log.61.zst
doesn't exist -- won't try to dispose of it

... instead of error renaming ... Read-only file system (as logrotate.service does)

Why does it work when running manually as root, but fails when it is executed by the logrotate.service?

I did not make any changes to logrotate.service. For completeness, here is the unit file:

$ systemctl cat logrotate.service
# /lib/systemd/system/logrotate.service
[Unit]
Description=Rotate log files
Documentation=man:logrotate(8) man:logrotate.conf(5)
ConditionACPower=true

[Service]
Type=oneshot
ExecStart=/usr/sbin/logrotate /etc/logrotate.conf

# performance options
Nice=19
IOSchedulingClass=best-effort
IOSchedulingPriority=7

# hardening options
#  details: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
#  no ProtectHome for userdir logs
#  no PrivateNetwork for mail deliviery
#  no ProtectKernelTunables for working SELinux with systemd older than 235
#  no MemoryDenyWriteExecute for gzip on i686
PrivateDevices=true
PrivateTmp=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectSystem=full
RestrictRealtime=true

Now I'm running out of options. Any help in troubleshooting the problem is much appreciated.

Improve this question
0

2 Answers 2

Reset to default
6

I think, I found it. The logrotate file is not the problem.

Instead, it is caused by the hardening features in the systemd unit file. After I disabled the ProtectSystem=full hardening options, it worked. The reason is that logrotate has to operate on the /usr directory in my case, which is readonly if the option is enabled.

From the documentation on ProtectSystem=

Takes a boolean argument or the special values "full" or "strict". If true, mounts the /usr and the boot loader directories (/boot and /efi) read-only for processes invoked by this unit. If set to "full", the /etc directory is mounted read-only, too. If set to "strict" the entire file system hierarchy is mounted read-only, except for the API file system subtrees /dev, /proc and /sys (protect these directories using PrivateDevices=, ProtectKernelTunables=, ProtectControlGroups=). This setting ensures that any modification of the vendor-supplied operating system (and optionally its configuration, and local mounts) is prohibited for the service. It is recommended to enable this setting for all long-running services, unless they are involved with system updates or need to modify the operating system in other ways. If this option is used, ReadWritePaths= may be used to exclude specific directories from being made read-only. This setting is implied if DynamicUser= is set. This setting cannot ensure protection in all cases. In general it has the same limitations as ReadOnlyPaths=, see below. Defaults to off.

To fix it properly, I added now the following line to /lib/systemd/system/logrotate.service:

ReadWritePaths=/usr/local/openresty/nginx/logs

Then ProtectSystem=full will mount everything as readonly except the directory where the logs are. Now, the readonly error is gone:

# systemctl daemon-reload && systemctl start logrotate
(completed successfully without output)

I assume, I never hit the problem on Ubuntu 18.04 because the hardening options were not yet the defaults. Ubuntu 20.04 has it now, but I don't know in what Ubuntu release they were first introduced.

Improve this answer
2
  • 1
    Thank you very much for the detailed answer. I was totally clueless as suddenly log rotation stopped and never raised any sort of error. Ultimately we hit a disk full problem to realize that the log file had grown to tens of GBs.. !
    – brahmana
    Jan 16 at 13:18
  • 1
    Thanks, I was facing exactly the same problem. :)
    – Flo
    Apr 13 at 15:41
1

Nice find to the same problem i was seeing. Editing nginx.conf to log to different directory (eg. the common /var/log/nginx) is an alternative that also works.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.