We are using:
root@server:~# /usr/share/logstash/bin/logstash --version
logstash 6.2.4
on:
root@server:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial
with the following log4j2.properties configuration:
status = error
name = LogstashPropertiesConfig
appender.rolling.type = RollingFile
appender.rolling.name = plain_rolling
appender.rolling.fileName = ${sys:ls.logs}/logstash-${sys:ls.log.format}.log
appender.rolling.filePattern = ${sys:ls.logs}/logstash-${sys:ls.log.format}-%d{yyyy-MM-dd}.log.gz
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.action.type = Delete
appender.rolling.strategy.action.basePath = ${sys:ls.logs}
appender.rolling.strategy.action.condition.type = IfLastModified
appender.rolling.strategy.action.condition.age = 7D
appender.rolling.strategy.action.pathConditions.type = IfFileName
appender.rolling.strategy.action.pathConditions.glob = logstash-${sys:ls.log.format}-*
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %-.10000m%n
appender.json_rolling.type = RollingFile
appender.json_rolling.name = json_rolling
appender.json_rolling.fileName = ${sys:ls.logs}/logstash-${sys:ls.log.format}.log
appender.json_rolling.filePattern = ${sys:ls.logs}/logstash-${sys:ls.log.format}-%d{yyyy-MM-dd}.log
appender.json_rolling.policies.type = Policies
appender.json_rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.json_rolling.policies.time.interval = 1
appender.json_rolling.policies.time.modulate = true
appender.json_rolling.layout.type = JSONLayout
appender.json_rolling.layout.compact = true
appender.json_rolling.layout.eventEol = true
rootLogger.level = ${sys:ls.log.level}
rootLogger.appenderRef.rolling.ref = ${sys:ls.log.format}_rolling
According to the Apache Log4j 2 documentation for RollingFileAppender:
If the file pattern ends with ".gz", ".zip", ".bz2", ".deflate", ".pack200", or ".xz" the resulting archive will be compressed using the compression scheme that matches the suffix.
We have set our filePattern to:
${sys:ls.logs}/logstash-${sys:ls.log.format}-%d{yyyy-MM-dd}.log.gz
but are finding that the rolled over logs are labelled with the .gz extensions, but are in fact, not compressed, e.g.:
root@server:~# file /var/log/logstash/logstash-plain-2019-01-17.log.gz
/var/log/logstash/logstash-plain-2019-01-17.log.gz: ASCII text
We have gzip installed. What have we missed? Any help would be appreciated.
gzipprogram is not required for Log4J 2 to compress log files - it does it using an internal Java implementation. Please note that it will only compress files after they have been rolled - the current file is never compressed. Also, if I understand correctly, Log4J 2 will not compress old files if the application terminated and restarted - files will only be compressed if the trigger condition (7 days in your case) triggered while the application was running. Finally - this is not an Ubuntu specific questions, you may want to try it in Stack Overflow.logstash-plain.logand/var/log/logstash/logstash-plain-2019-01-17.log.gzis the archived log. We don't want the current log to be compressed, just the archived ones. Are we to understand that the archived logs will be compressed after 7 days? I would be more than happy to migrate the question to SO but unfortunately, my reputation isn't sufficient for this task.appender.rolling.strategy.action.condition.age = 7D.