5

I have a cronjob that runs some Python scripts and they generate massive log file. How do I prevent them from getting so big? It'd be perfect if the file was deleted once it hit a certain size. Because every few days, the file is 50MB now but gets much bigger and takes forever to download and analyze a bug on a slow internet connection. I really only want to see that last 1000 lines to see where the script went wrong.

I know I could code something to do this, but I was just wondering if Linux had something that I didn't know about. I imagine big log files are a common problem. Thank-you.

1
  • 1
    I think the program is called logrotate (and is installed by default in Ubuntu). I don't have familiarity with configuring it, though.
    – edwin
    Mar 22 '14 at 2:17
8

Usually you should only log the activity you care about and should configure the program to be less verbose if possible.

As suggested, using logrotate is the usual way of dealing with logfiles. But instead of adding content to /etc/logrotate.conf you should add your own job to /etc/logrotate.d/, otherwise you would have to look at more diffs of configuration files during release upgrades.

Here is an example I created for a php script:

/var/log/some-php-app/*.log {
        daily
        rotate 10
        delaycompress
        compress
        notifempty
        missingok
}
  • /var/log/some-php-app/*.log - Is the path where the log file(s) is located and it's name. As you can see, you can also use wildcards to apply the rule to more than one logfile.
  • daily - Rotates logfiles daily. Alternatives: hourly, weekly, monthly, yearly, maxsize, maxage
  • rotate count - Files are rotated count times before being removed or mailed to the address specified in a mail directive.
  • delaycompress - Postpones compression of the previous log file to the next rotation cycle.
  • compress - Compresses old versions of log files with gzip. You can use zless instead of less to view the files.
  • notifempty - Does not rotate the log if it is empty.
  • missingok - Goes on to the next rotate job without issuing an error message, when the log file is missing.

Have a look at the logrotate manpage for more options.

1
  • You can test it by doing a dry run with sudo logrotate /etc/logrotate.conf --debug
    – Ricardo
    Nov 19 '20 at 18:30
2

You will want to set a size in /etc/logrotate.conf

The file exists by default and is ran daily. The only thing you will need to do is add some additional parameters to logrotate.conf.

/path/to_big.log {
        size 1k
        create 700 user group
        rotate 1
}

In this example we specify the path to our log,the size we want to limit it to, the permissions for the replacement log, and how many copies to keep. Rotate 1 will keep one copy.

2

If your logs are growing very fast you can't wait for cron to run logrotate once a day as is the default after setting the size limit as mentioned in the previous answer, you can make a symlink in /etc/cron.hourly to the logrotate cron job in /etc/cron.daily with this command:

sudo ln -s /etc/cron.daily/logrotate /etc/cron.hourly/

The only thing you need to remember, is to be sure and delete the link once the problem that is creating the excessive logs is resolved. That is if you have something outside of your control creating excessive logs, you need to report a bug about the problem, because it may be affecting others without realizing it. This kind of bug can cause the root volume to run out of space and then people will see all kinds of problems and never realize that it is because they are running out of space on root.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.