It is important to control the sizes of log files on a Linux server because
their size always grows over time. Every server has limited resources and too
large logs can lead to performance and memory problems, not to mention the loss
of precious storage space. This problem is typically solved through log
rotation, a process that involves renaming or compressing a log file before it
gets too large, and cleaning up old logs to reclaim storage. The logrotate
program is the weapon of choice for log rotation in most Linux distributions,
and it's what we'll be working with through out this tutorial.
By reading through this article, you will learn how to do the following:
View and edit the configuration for the `logrotate` utility.
Learn the differences between the general and application-specific configuration.
Create a standard `logrotate` configuration for your application.
Create a system-independent log rotation schedule for your application.
Prerequisites
Ensure that you've met the following prerequisites before proceeding with the
rest of this tutorial:
Basic knowledge of working with the Linux command line.
A Linux server that includes the non-root user with sudo access. We'll be
using Ubuntu 20.04 in this guide.
The logrotate daemon is pre-installed and active by default in Ubuntu. This
daemon uses configuration files that specify all the log rotation details for an
application. The default setup consists of the following aspects:
/etc/logrotate.conf: this is the general configuration file for logrotate.
/etc/logrotate.d: this directory includes files that configure a specific
application.
We will examine both configuration possibilities below.
The general configuration
First off, let's view the general configuration file at /etc/logrotate.conf.
Go ahead and print its contents with the cat utility:
Copied!
cat /etc/logrotate.conf
The command above prints the entire contents of this file:
Output
# see "man logrotate" for details
# rotate log files weekly
weekly
# use the adm group by default, since this is the owning group
# of /var/log/syslog.
su root adm
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
#dateext
# uncomment this if you want your log files compressed
#compress
# packages drop log rotation information into this directory
include /etc/logrotate.d
# system-specific logs may be also be configured here.
Here's a description of what each configuration directive means:
weekly: logs are rotated every week. Alternatively, you can specify another
time interval (daily, weekly, monthly, or yearly). You can also rotate
the logs every hour (hourly), but note that you'll need to set up a cron job
if a shorter rotation frequency than daily is desired.
su root adm: log rotation is performed with the root user and admin group.
rotate 4: log files are rotated four times before old files are removed. If
rotate is set to zero, then the old versions are removed immediately.
create: immediately after rotation, create a new log file with the same name
as the one just rotated.
compress: this rule determines whether old log files should be compressed or
not. Log compression is turned off by default.
include: this specifies the directory for application-specific
configuration.
Application-specific configuration
Let's view the contents of the /etc/logrotate.d directory that contains
application-specific configuration for logrotate.
Copied!
ls /etc/logrotate.d/
The command above lists all files in this directory:
You can observe that quite a few programs have their configuration files in this
directory. For example, let's take a look at the first 15 files in the config
file for rsyslog through the head command.
Copied!
head -n 15 /etc/logrotate.d/rsyslog
You'll see the program's output appear on the screen:
The output shows that the /var/log/syslog file is rotated daily and keeps
seven compressed backups. It also includes the following directives:
missingok: do not report any error if the log file is missing.
notifempty: do not rotate the log if it is empty.
delaycompress: postpone compression of the previous log file to the next
rotation cycle when a program is using it.
postrotate/endscript: the lines between postrotate and endscript are
executed after the log file is rotated.
A Logrotate configuration may include various other directives. For example, you
can specify that older log files are uploaded to another server or archived, or
you can rotate a log file when it exceeds a predefined size. All possible
directives are described in logrotate manual pages which can be examined by
executing man logrotate.
Step 2 — Creating a log file for a custom application
In this step, we will create a log file for a fictional custom application
called my-custom-app. In subsequent sections, we will set up a log rotation
policy for the logs produced by this application.
Create a new subdirectory in /var/log with mkdir:
Copied!
sudo mkdir /var/log/my-custom-app
The mkdir command creates a new directory where we will place our custom log
file.
Go ahead and create a log file called backup.log in the new my-custom-app
directory, and open it in your text editor.
Note that the exact contents of this file is not important for the purpose of
this tutorial. In a real-world scenario, the log entries in the file will be
produced by the application itself.
Save and close the file, then head to the next section to learn how to configure
a log rotation policy for this application's logs.
Step 3 — Creating a standard Logrotate configuration
In this step, we will create a standard logrotate job for our application's
logs. After we set up a new logrotate configuration file, it will be executed
with all the other system log rotation jobs once a day. We will use the
my-custom-app example from the previous step.
Create a new my-custom-app file in the /etc/logrotate.d directory with your
text editor:
The configuration above applies to all the files ending with .log in the
/var/log/my-custom-app directory. We've already discussed what each directive
means in step 1, so we won't go over that again.
Let's go ahead and test the new configuration by executing the logrotate
command. The --debug option ensures that logrotate operates in test mode
where only debug messages are printed.
Copied!
sudo logrotate /etc/logrotate.conf --debug
The logrotate program reads the configuration file at /etc/logrotate.conf
and prints some messages. You should spot an entry for the my-custom-app
configuration that looks similar to what is displayed below:
Output
. . .
rotating pattern: /var/log/my-custom-app/*.log after 1 days (7 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 4
considering log /var/log/my-custom-app/backup.log
Creating new state
Now: 2021-04-07 12:27
Last rotated at 2021-04-07 12:00
log does not need rotating (log has been already rotated)
switching euid to 0 and egid to 0
. . .
The above output indicates that the new configuration file at
/etc/logrotate.d/my-custom-app has been found by the logrotate program.
Therefore, the log files in the my-custom-app directory will now be rotated
according to the policy defined within it along with the other system and
application logs.
Step 4 — Change the ownership of the log directory
An alternative to the approach taken in the previous step involves creating a
configuration file that is system-independent. Such a configuration will not be
included in the /etc/logrotate.d/ directory. Instead, we will create a cron
job that will execute the configuration file at custom time interval.
There are two main reasons for using a system-independent configuration:
The logrotate program cannot rotate logs at an interval of less than once
per day with the standard configuration.
The standard configuration cannot rotate logs as a non-root user due to
directory permissions.
Similar to the previous step, we will use the my-custom-app example to
demonstrate a system-independent configuration.
First off, let's confirm the owner of its log files with the ls command (the
-l option shows access rights to the file):
Copied!
ls -l /var/log/my-custom-app/backup.log
The program's output should appear on the screen:
Output
-rw-r--r-- 1 root root 134 Feb 10 09:16 /var/log/my-custom-app/backup.log
The output shows that the owner of this log file is root. Let's go ahead and
change the ownership of the my-custom-app directory and all its contents to
the current user instead of the root user so that we'll be able to modify them
without root privileges. Replace the <username> placeholder below with the
current logged in user:
Copied!
sudo chown -R <username>: /var/log/my-custom-app
Afterward, execute the previous ls command again.
Copied!
ls -l /var/log/my-custom-app/backup.log
You'll notice that the ownership has changed (ayo is the username on my
server):
Output
-rw-r--r-- 1 ayo ayo 134 Feb 10 09:16 /var/log/my-custom-app/backup.log
Step 5 — Creating a custom Logrotate configuration
In this step, we'll create a custom Logrotate configuration in the user's home
directory. Go ahead and create it with your text editor:
Copied!
nano /home/<user>/logrotate.conf
Set up a logrotate configuration for the my-custom-app application by adding
the following lines to the file:
We've defined that the above configuration should apply to all files in the
/var/log/my-custom-app directory that end with the .log extension. Note that
the time interval is specified as hourly which is not supported in the
standard configuration.
Logrotate does not know about this new configuration because it is not included
in the /etc/logrotate.conf file. So instead of adding the new configuration
there, we will use a logrotate state file. This file records what the
logrotate did last time it ran, and it informs the program of what to do the
next time it runs.
The default status file for Logrotate is located in /var/lib/logrotate/status.
However, we will create a custom one through the command below:
The --state option tells logrotate to use an alternative state file located
at /home/<username>/custom-state. The logrotate CLI will create this file if
it doesn't exist. You can view its content with cat:
Copied!
cat /home/<username>/custom-state
You'll see the program's output appear on the screen:
Output
logrotate state -- version 2
"/var/log/my-custom-app/backup.log" 2022-2-10-10:0:0
The output indicates that logrotate identified the backup.log file in the
/var/log/my-custom-app directory. Notice that we no longer need root access to
execute the logrotate command. That being said, it won't run again unless we
intervene manually. Therefore, in the last step for this tutorial, we will
create a cron job that executes the command periodically after some time has
passed (every hour to be specific).
Step 6 — Running Logrotate periodically with Cron
Go ahead and open the cron jobs configuration file by executing crontab -e in
your terminal:
Copied!
crontab -e
The -e option is used to edit the current user's cron jobs using the editor
specified by the $VISUAL or $EDITOR environmental variables. The above
command should open a configuration file in your preferred text editor specified
by these variables.
Go to the bottom of the file and add the following line:
This new line specifies that the cron job will be executed every hour (at
minute 0) and the logrotate command will run with our custom configuration and
state file.
Save and close the modified file. You will observe the following output:
Output
crontab: installing new crontab
Now that your system-independent log rotation policy is all set up, you can view
the /var/log/my-custom-app directory after an hour to confirm that the log
files therein are rotated according to the defined policy. For more details
about cron jobs see the following
documentation or type
man cron in your terminal.
Conclusion and next steps
In this tutorial, you learned about log rotation in Linux and how it is achieved
through the logrotate program. We started by examining its configuration
files, and then we discussed several of the common directives you are likely to
encounter. We also created a standard logrotate configuration for our custom
application, before pivoting to one that is system-independent.
To learn more about logrotate and everything that you can do with it, do check
out its manual page by running man logrotate in your terminal. Thanks for
reading, and happy logging!
Article by
Ayooluwa Isaiah
Ayo is a technical content manager at Better Stack. His passion is simplifying and communicating complex technical ideas effectively. His work was featured on several esteemed publications including LWN.net, Digital Ocean, and CSS-Tricks. When he's not writing or coding, he loves to travel, bike, and play tennis.
Are you a developer and love writing and sharing your knowledge with the world? Join our guest
writing program and get paid for writing amazing technical guides. We'll get them to the right
readers that will appreciate them.