4

I'm using Ubuntu 14.04. I have the following in my /etc/logrotate.conf file ...

/home/rails/myproject/log {
        daily
        rotate 3
        compress
        delaycompress
        missingok
        notifempty
        create 644 rails rails
}

/var/log/postgresql {
        daily
        rotate 3
        compress
        delaycompress
        missingok
        notifempty
        create 644 root root
}

Every night, I would look at my rails logs and it would always be bigger -- i.e. it didn't seem like the logs were getting rotated ...

myuser@myproject:~$ ls -al /home/rails/myproject/log
total 4574368
drwxr-xr-x  2 rails rails       4096 May 30 12:04 .
drwxr-xr-x 15 rails rails       4096 May 30 12:03 ..
-rw-rw-r--  1 rails rails      14960 Jun  1 22:39 development.log
-rw-rw-r--  1 rails rails          0 Oct 22  2016 .keep
-rw-r--r--  1 rails rails 4523480004 Jun 22 10:19 production.log
-rw-rw-r--  1 rails rails  156358087 Jun 22 10:19 sidekiq.log
-rw-rw-r--  1 rails rails      54246 Apr 10 14:34 test.log

When I run the command manually, I see that some of the logs seem to get rotated ...

myuser@myproject:~$ sudo logrotate /etc/logrotate.conf
myuser@myproject:~$ ls -al /home/rails/myproject/log
total 4570288
drwxr-xr-x  2 rails rails       4096 Jun 22 10:22 .
drwxr-xr-x 15 rails rails       4096 May 30 12:03 ..
-rw-rw-r--  1 rails rails          0 Jun 22 10:22 development.log
-rw-rw-r--  1 rails rails      14960 Jun  1 22:39 development.log.1
-rw-rw-r--  1 rails rails          0 Oct 22  2016 .keep
-rw-r--r--  1 rails rails          0 Jun 22 10:22 production.log
-rw-r--r--  1 rails rails 4523505906 Jun 22 10:23 production.log.1
-rw-rw-r--  1 rails rails  156369048 Jun 22 10:23 sidekiq.log
-rw-rw-r--  1 rails rails      54246 Apr 10 14:34 test.log

How do I figure out why my rails logs are not rotated nightly? Note that other logs in the system seem to be. Above, I included my postgres configuration, and when I look at the logs there, seem to be rotating normally ...

myuser@myproject:~$ ls -al /var/log/postgresql
total 1832
drwxrwxr-t  2 root     postgres    4096 May  2 20:42 .
drwxr-xr-x 13 root     root        4096 Jun 22 10:22 ..
-rw-r-----  1 postgres adm      1861361 Jun 22 10:14 postgresql-9.6-main.log

Thanks, - Dave

Edit: Putting the configuration in a separate file didn't seem to do anything. Below is my configuration and also the logs that didn't appear to get rotated ...

myuser@myapp:~$ sudo cat /etc/logrotate.d/myapp
[sudo] password for myuser:
/home/rails/myapp/log/*.log {
   daily
   missingok
   compress
   notifempty
   rotate 12
   create
   delaycompress
   missingok
   su rails rails
}

Here are the logs. Doesn't appear anything happened ...

myuser@myapp:~$ ls -al /home/rails/myapp/log
total 4635956
drwxr-xr-x  2 rails rails       4096 Jun 22 10:22 .
drwxr-xr-x 15 rails rails       4096 May 30 12:03 ..
-rw-rw-r--  1 rails rails          0 Jun 22 10:22 development.log
-rw-rw-r--  1 rails rails      14960 Jun  1 22:39 development.log.1
-rw-rw-r--  1 rails rails          0 Oct 22  2016 .keep
-rw-r--r--  1 rails rails          0 Jun 22 10:22 production.log
-rw-r--r--  1 rails rails 4546785231 Jun 24 12:12 production.log.1
-rw-rw-r--  1 rails rails  200336693 Jun 24 12:51 sidekiq.log
-rw-rw-r--  1 rails rails      54246 Apr 10 14:34 test.log
  • The postgresql log files don't seem to be rotated from the output you've included in the question. Are you sure they're being rotated correctly? It's also recommended to use logrotate configuration files within /etc/logrotate.d rather than using /etc/logrotate.conf, does logrotate.conf still have its original content too? – Arronical Jun 22 '17 at 15:11
  • Why do you say the postgres logs don't seem to be rotated? If they wren't rotated wouldn't I have one continuously growing file? – Dave Jun 22 '17 at 15:48
  • That looks like what you have, the . and .. files refer to the directory /var/log/postgresql, and the parent directory /var/log, you see these because you're using the -a switch on ls. If they were rotating you would have files postgrsql-9.6-main.log, postgrsql-9.6-main.log.1 and postgrsql-9.6-main.log.2.gz. – Arronical Jun 22 '17 at 15:52
  • Ok, so its your advice to remove configuration from /etc/logrotate.conf and instead create files within /etc/logrotate.d/ ? I dind't think it mattered where the configuration was located. – Dave Jun 22 '17 at 16:03
  • It doesn't matter from logrotate's perspective, but there is a load of stuff in /etc/logrotate.conf that needs to be there, like rotating /var/log/wtmp and /var/log/btmp. I've just noticed the section labelled #system-specific logs... so suppose there's no harm in appending logrotate instructions. I find it easier having specific files, for ease of management. The answer below should solve the problem I think you're having. – Arronical Jun 22 '17 at 16:09
0
+50

Logrotate's job is to move (rename), and compress, files. You've configured it in this case to rename and compress the Rails log files, then create new ones with the original names.

File names are a way of finding a file, but the actual file is just some space on disk. A file can have multiple names (hard links) or no names (you can rm a file that's open, but it still takes up space on disk as long as the file is open in some process).

The problem you seem to be having here is that the Rails app has already got the file open when it gets renamed. The Rails logger doesn't notice the name change, because it's used the name once to open the file, and then completely stopped caring about its name. The logger just has a handle on the open file.

You have to persuade it to close and re-open the log files, using their names again, meaning it starts writing to the new empty ones which now have the name the old ones used to have.

If you have a look in /etc/logrotate.d you'll see a lot of examples of this, depending what you've got installed.

For example, rsyslog has:

postrotate
    invoke-rc.d rsyslog rotate > /dev/null
endscript

stunnel has:

postrotate
    /etc/init.d/stunnel4 reopen-logs > /dev/null
endscript

These are scripts to tell the relevant process that the file needs re-opening. The specific mechanism depends on the program, but what this tends to be doing is sending a HUP (or sometimes USR1) (see man 7 signal), which long-running processes take as an instruction to close and re-open logfiles.

In the case of Rails the way of doing it varies depending on what logger you're using. I've just seen some advice suggesting you should use copytruncate which is basically a "cheat option" in logrotate telling it to manually copy the contents out and empty the file rather than moving it and making a new one. (see man logrotate.conf). This is used instead of create like so:

/home/rails/myapp/log/*.log {
    daily
    missingok
    compress
    notifempty
    rotate 12
    copytruncate
    delaycompress
    missingok
    su rails rails
}

This is not a great solution as it is literally copying the whole file (to create a snapshot of it as it is) before deleting its contents, which is quite inefficient.

However, if you're using Unicorn to run your app (which multiplexes requests across a bunch of identical Rails worker processes) it supports the USR1 signal as normal (killing and replacing all the workers, effectively causing them to re-open the files) and you can just send it in a postrotate using pkill or similar, perhaps like this:

/home/rails/myapp/log/*.log {
    daily
    missingok
    compress
    notifempty
    rotate 12
    create
    delaycompress
    missingok
    su rails rails
    postrotate
        pkill -USR1 -u rails unicorn
    endscript
}

pkill is a tool to search running processes and send them signals, so this would find everything with the name unicorn running as the rails user and send it the USR1 signal that tells it to re-open the log files. (The examples I gave from Ubuntu packages' /etc/logrotate.d files are actually doing the same thing but those services have the searching hidden in functions in their /etc/init.d scripts.)

I'm sure there'll be some way to configure a sensible postrotate for whatever Rails setup you've got (in the worst and easiest case, just restart it), but hopefully that explains the Ubuntu side of things anyway...

  • I'm still a novice at all this and I'm not understanding what you're explaning. You're saying the problem is on the Rails side and not with my logrotate? Just for kicks, what is the configuration I would you for this cheating option you mention? As long as it gets the job done, I'm happy to cheat. – Dave Jul 3 '17 at 3:21
  • I've tried to clarify a bit and also given an example of copytruncate. It's just a different (less efficient) method that replaces create, designed for processes that can't re-open files properly. – Tom Spurling Jul 3 '17 at 10:58
  • Thanks for these clarifications. Is there any way I can test these out before waiting to see if they execute tonight? I don't want to leave you hanging on the bounty if this indeed solves my problem. – Dave Jul 3 '17 at 20:05
  • I'm a bit late but yes, you can trigger an early rotation for any specific config file: sudo logrotate -fv /etc/logrotate.d/rails for example. -f means "ignore the fact the file's not old enough and just do it". -v means "explain what you're doing". (There's alternatively a -d which means "say what you would do, without doing it", although in this case that's not really helpful as you're more interested in whether the Rails app behaves after the rotation.) Good luck! – Tom Spurling Jul 4 '17 at 1:44
  • This reminds me, this is probably the main reason to use separate config files for each application (apart from having to split it across packages). If the stuff you were wanting to test was in /etc/logrotate.conf, your test run would end up doing everything because that says include /etc/logrotate.d. – Tom Spurling Jul 4 '17 at 1:51
1

It looks like the problem comes down to you specifying a directory for file rotation, instead of actual file names. The configuration files for logrotate accept wildcards for globbing (pattern matching).

To rotate all files with a .log extension in your /home/rails/myproject/log directory you could use the following line in place of the first line of your configuration:

/home/rails/myproject/log/*.log {

and similarly in the postgres directory configuration

/var/log/postgresql/*.log {

It's possible to use the * wildcard without the .log extension, to rotate all files (except hidden ones starting with a .) in your postgresql directory, but I prefer the added control of specifying .log files only:

/var/log/postgresql/* {

As a side note, be aware of how you create new versions of the log file with logrotate, if you create the new postgresql log with 644 octal permissions, owned by the root user, then the postgres user will not be able to write into the new log file.

  • I'm more concerned about the Rails logs since they seem to keep growing so let me apply the settings you have here and see how things shake out tonight. - Dave – Dave Jun 22 '17 at 18:32
  • You may need to wait until a second day has passed for the logs to rotate if you had run a logrotate manually yesterday. I seem to remember something about it being fussy about a full 24 hours having passed. Fingers crossed! – Arronical Jun 23 '17 at 8:37
  • Oh shoot, I didn't see your comment. It didn't appear things had rotated last night, so I'll wait for tonight and post some output tomorrow. Thanks – Dave Jun 23 '17 at 15:32
  • Hi, Ok, I edited my question to include what the logs looked like after the overnight run. They are the same. It doesn't look like anyting got rotated. Is there any logs of waht /etc/logrotate does? I'm curious why it doesn't appear to be rotating anything. – Dave Jun 24 '17 at 16:57
0

Check the status in /var/lib/logrotate/status if showing any issues

Please check the permissions and ownership of files in /etc/logrotate.d root owner and permission mode 644. code snippet for logrotate:

/home/rails/myapp/log/*.log {
   rotate 12
   daily
   missingok
   compress
   notifempty
   create 640 rails rails
   delaycompress
   missingok
}

do check the logrotate output by manual execution with --verbose If you require logrotation for more than a spefici size you can experiment with maxsize and size options in logrotate config file

  • My permissions and ownership for the configuration file are "-rw-r--r-- 1 root root 161 Jun 22 12:04 rails_config", which is the same as all the other configurations. I assume this was the permissions you were referring to? – Dave Jul 3 '17 at 20:03
  • Yes, now check last night's log rotate status. There will be some clue related to rotation or skipping the specific file due to some reason – v_sukt Jul 4 '17 at 3:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.