1

We are using:

root@server:~# /usr/share/logstash/bin/logstash --version
logstash 6.2.4

on:

root@server:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:    16.04
Codename:   xenial

with the following log4j2.properties configuration:

status = error
name = LogstashPropertiesConfig

appender.rolling.type = RollingFile
appender.rolling.name = plain_rolling
appender.rolling.fileName = ${sys:ls.logs}/logstash-${sys:ls.log.format}.log
appender.rolling.filePattern = ${sys:ls.logs}/logstash-${sys:ls.log.format}-%d{yyyy-MM-dd}.log.gz
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.action.type = Delete
appender.rolling.strategy.action.basePath = ${sys:ls.logs}
appender.rolling.strategy.action.condition.type = IfLastModified
appender.rolling.strategy.action.condition.age = 7D
appender.rolling.strategy.action.pathConditions.type = IfFileName
appender.rolling.strategy.action.pathConditions.glob = logstash-${sys:ls.log.format}-*
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %-.10000m%n

appender.json_rolling.type = RollingFile
appender.json_rolling.name = json_rolling
appender.json_rolling.fileName = ${sys:ls.logs}/logstash-${sys:ls.log.format}.log
appender.json_rolling.filePattern = ${sys:ls.logs}/logstash-${sys:ls.log.format}-%d{yyyy-MM-dd}.log
appender.json_rolling.policies.type = Policies
appender.json_rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.json_rolling.policies.time.interval = 1
appender.json_rolling.policies.time.modulate = true
appender.json_rolling.layout.type = JSONLayout
appender.json_rolling.layout.compact = true
appender.json_rolling.layout.eventEol = true

rootLogger.level = ${sys:ls.log.level}
rootLogger.appenderRef.rolling.ref = ${sys:ls.log.format}_rolling

According to the Apache Log4j 2 documentation for RollingFileAppender:

If the file pattern ends with ".gz", ".zip", ".bz2", ".deflate", ".pack200", or ".xz" the resulting archive will be compressed using the compression scheme that matches the suffix.

We have set our filePattern to:

${sys:ls.logs}/logstash-${sys:ls.log.format}-%d{yyyy-MM-dd}.log.gz

but are finding that the rolled over logs are labelled with the .gz extensions, but are in fact, not compressed, e.g.:

root@server:~# file /var/log/logstash/logstash-plain-2019-01-17.log.gz 
/var/log/logstash/logstash-plain-2019-01-17.log.gz: ASCII text

We have gzip installed. What have we missed? Any help would be appreciated.

  • The Linux gzip program is not required for Log4J 2 to compress log files - it does it using an internal Java implementation. Please note that it will only compress files after they have been rolled - the current file is never compressed. Also, if I understand correctly, Log4J 2 will not compress old files if the application terminated and restarted - files will only be compressed if the trigger condition (7 days in your case) triggered while the application was running. Finally - this is not an Ubuntu specific questions, you may want to try it in Stack Overflow. – Guss Jan 18 at 10:34
  • @Guss, yes, the current log is logstash-plain.log and /var/log/logstash/logstash-plain-2019-01-17.log.gz is the archived log. We don't want the current log to be compressed, just the archived ones. Are we to understand that the archived logs will be compressed after 7 days? I would be more than happy to migrate the question to SO but unfortunately, my reputation isn't sufficient for this task. – dw8547 Jan 18 at 10:50
  • @Guss please make sure you know a site's scope before suggesting migration. First, this is completely on topic here, as long as the OP is using Ubuntu. Note that there is essentially nothing that is "specific" to Ubuntu and so we do not (and cannot) require that a question be Ubuntu-specific to be on topic. What's more, this question would be completely off topic on Stack Overflow since it has nothing to do with programming. – terdon Jan 20 at 13:03
  • Regarding my own comment from above ("Are we to understand that the archived logs will be compressed after 7 days?"), of course they will not because they will be deleted after 7 days as per appender.rolling.strategy.action.condition.age = 7D. – dw8547 Jan 24 at 8:31

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.