At the time of writing, the Apache HTTP server is used by 30.8% of all web servers in operation. If you're responsible for managing any system that utilizes Apache, then you will surely interact with its logging infrastructure on a regular basis. This tutorial will introduce you to logging in Apache and how it can help you diagnose, troubleshoot, and quickly resolve any problem you may encounter on your server.
You will learn where logs are stored, how to access them, and how to customize the log output and location to fit your needs. You will also learn how to centralize Apache logs in a log management system for easier tracing, searching, and filtering of logs across your entire stack.
Here's an outline of what you will learn by following through with this tutorial:
To follow through with this tutorial, you should set up a Linux server that
includes a non-root user with sudo privileges. Additionally, you also need the
Apache HTTP server installed and enabled on the server, which can be done by
executing the relevant commands below.
On Debian-based distributions like Ubuntu:
sudo apt install apache2
sudo systemctl enable apache2
sudo systemctl start apache2
On RHEL, Fedora or CentOS:
sudo dnf install httpd
sudo systemctl enable httpd
sudo systemctl start httpd
Please note that the rest of the commands, directory configurations, and conventions used in this tutorial pertain to Debian-based distributions like Ubuntu. Still, the concepts remain the same for other distributions.
Apache logs are files that record everything the Apache web server is doing for later analysis by the server administrator. The records of all Apache events are placed in two different text files:
The log files' location depends on the operating system the Apache web server is
running. On Debian-based operating systems like Ubuntu, the access log file is
located in /var/log/apache2/access.log. On CentOS, RHEL, or Fedora, the access
log file is stored in /var/log/httpd/access_log.
A typical access log entry might look like this:
::1 - - [13/Nov/2020:11:32:22 +0100] "GET / HTTP/1.1" 200 327 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36"
Similarly, the error log file is located in /var/log/apache2/error.log on
Debian-based systems and /var/log/httpd/error_log on CentOS, RHEL, or Fedora.
A typical error log entry might look like this:
[Thu May 06 12:03:28.470305 2021] [php7:error] [pid 731] [client ::1:51092] script '/var/www/html/missing.php' not found or unable to stat
In the next section, we'll discuss how to view these log files from the command line.
One of the most common ways to view an Apache log file is through the tail
command which prints the last 10 lines from a file. When the -f option is
supplied, the command will watch the file and output its contents in real-time.
sudo tail -f /var/log/apache2/access.log
You should observe the following output on the screen:
. . .
198.54.132.137 - - [04/Feb/2022:11:34:04 +0000] "GET / HTTP/1.1" 200 3477 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
198.54.132.137 - - [04/Feb/2022:11:34:04 +0000] "GET / HTTP/1.1" 200 3477 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
198.54.132.137 - - [04/Feb/2022:11:34:04 +0000] "GET / HTTP/1.1" 200 3477 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
198.54.132.137 - - [04/Feb/2022:11:34:05 +0000] "GET / HTTP/1.1" 200 3476 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
198.54.132.137 - - [04/Feb/2022:11:34:06 +0000] "GET / HTTP/1.1" 200 3476 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
198.54.132.137 - - [04/Feb/2022:11:34:06 +0000] "GET / HTTP/1.1" 200 3476 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
198.54.132.137 - - [04/Feb/2022:11:34:07 +0000] "GET / HTTP/1.1" 200 3476 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
198.54.132.137 - - [04/Feb/2022:11:34:07 +0000] "GET / HTTP/1.1" 200 3476 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
To view the entire contents of the file, you can use the cat command or open
the file in a text editor like nano or vim:
cat /var/log/apache2/access.log
You may also want to filter the log entries in the log file by a specific term.
In such cases, you should use the grep command. The first argument to grep
is the term you want to search for, while the second is the log file that will
be searched. In example below, we are filtering all the lines that contain the
word GET:
sudo grep GET /var/log/apache2/access.log
This should present the following output:
. . .
198.54.132.137 - - [04/Feb/2022:11:34:04 +0000] "GET / HTTP/1.1" 200 3477 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
198.54.132.137 - - [04/Feb/2022:11:34:04 +0000] "GET / HTTP/1.1" 200 3477 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
198.54.132.137 - - [04/Feb/2022:11:34:05 +0000] "GET / HTTP/1.1" 200 3476 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
198.54.132.137 - - [04/Feb/2022:11:34:06 +0000] "GET / HTTP/1.1" 200 3476 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
198.54.132.137 - - [04/Feb/2022:11:34:06 +0000] "GET / HTTP/1.1" 200 3476 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
198.54.132.137 - - [04/Feb/2022:11:34:07 +0000] "GET / HTTP/1.1" 200 3476 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
198.54.132.137 - - [04/Feb/2022:11:34:07 +0000] "GET / HTTP/1.1" 200 3476 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
The access log records all requests that are processed by the server. You can see what resources are being requested, the status of each request, and how long it took to process their response. In this section, we'll dive deeper into how to customize the information that is displayed in this file.
Before you can derive value from reading a log file, you need to understand the
format that is being used for each of its entries. The CustomLog directive is
what controls the location and format of the Apache access log file. This
directive can be placed in the server configuration file
(/etc/apache2/apache2.conf) or in your virtual host entry. Note that defining
the same CustomLog directive in both files may cause problems.
Let's look at the common formats used in Apache access logs and what they mean.
The Common Log Format
is the standardized access log format format used by many web servers because it
is easy to read and understand. It is defined in the /etc/apache2/apache2.conf
configuration file through the LogFormat directive.
When you run the command below:
sudo grep common /etc/apache2/apache2.conf
You will observe the following output:
LogFormat "%h %l %u %t \"%r\" %>s %O" common
The line above defines the nickname common and associates it with a particular
log format string. A log entry produced by this format will look like this:
127.0.0.1 alice Alice [06/May/2021:11:26:42 +0200] "GET / HTTP/1.1" 200 3477
Here's an explanation of the information contained in the log message above:
%h -> 127.0.0.1: the hostname or IP address of the client that made the
request.%l -> alice: remote log name (name used to log in a user). A placeholder
value (-) will be used if it is not set.%u -> Alice: remote username (username of logged-in user). A placeholder
value (-) will be used if it is not set.%t -> [06/May/2021:11:26:42 +0200]: the day and time of the request.\"%r\" -> "GET / HTTP/1.1" - the request method, route, and protocol.%>s -> 200 - the response code.%O -> 3477 - the size of the response in bytes.The Combined Log Format is very similar to the Common log format but contains few extra pieces of information.
It's also defined in the /etc/apache2/apache2.conf configuration file:
sudo grep -w combined /etc/apache2/apache2.conf
You will observe the following output:
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
Notice that it is exactly the same as the Common Log Format, with the addition of two extra fields. Entries produced in this format will look like this:
127.0.0.1 alice Alice [06/May/2021:11:18:36 +0200] "GET / HTTP/1.1" 200 3477 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"
Here's an explanation of the two additional fields that are not present in the Common log format:
\"%{Referer}i\" -> "-": the URL of the referrer (if available, otherwise
- is used).\"%{User-Agent}i\" ->
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36":
detailed information about the user agent of the client that made the request.You can define a custom log format in the /etc/apache2/apache2.conf file by
using the LogFormat directive followed by the actual format of the output and
a nickname that will be used as an identifier for the format. After defining the
custom format, you'll pass its nickname to the CustomLog directive and restart
the apache2 service.
In this example, we will create a log format named custom that looks like
this:
LogFormat "%t %H %m %U %q %I %>s %O %{ms}T" custom
Open your /etc/apache2/apache2.conf file and place the line above below the
other LogFormat lines. It will produce access log entries with the following
details:
%t: date and time of the request.%H: the request protocol.%m: the request method.%U: the URL path requested.%q: query parameters (if any).%I: total bytes received including the request headers.%>s: final HTTP status code.%O: number of bytes sent in the response.%{ms}T: time taken to generate the response in milliseconds.You can find all other formatting options and their description on this page.
To enable the custom format for subsequent access log entries, you must change
the value of the CustomLog directive in your virtual hosts file and restart
the apache2 service with Systemctl.
Open up the default virtual hosts file using the command below:
sudo nano /etc/apache2/sites-available/000-default.conf
Find the following line:
CustomLog ${APACHE_LOG_DIR}/access.log combined
And change it to:
CustomLog ${APACHE_LOG_DIR}/access.log combined
Save the file by pressing Ctrl-O then Ctrl-X, then restart the apache2
service using the command below:
sudo systemctl restart apache2
Afterward, make the following request to your server using curl:
curl --head 'http://<your_server_ip>?name=john&age=30'
You should observe the following response:
HTTP/1.1 200 OK
Date: Mon, 07 Feb 2022 14:21:45 GMT
Server: Apache/2.4.41 (Ubuntu)
Last-Modified: Mon, 07 Feb 2022 12:57:29 GMT
ETag: "2aa6-5d76d24a738bc"
Accept-Ranges: bytes
Content-Length: 10918
Vary: Accept-Encoding
Content-Type: text/html
Go ahead and view the last 10 messages in the access log file:
sudo tail /var/log/apache2/access.log
The log entry that describes the request will look like this:
[07/Feb/2022:14:21:45 +0000] HTTP/1.1 HEAD /index.html ?name=john&age=30 96 200 255 0
It's also possible to create multiple access log files by specifying the
CustomLog directive more than once. In the example below, the first line logs
into a custom.log file using the custom log format, while the second uses
the common format to write entries into access.log. Similarly, the
combined.log file contains messages formatted according to the combined log
format.
CustomLog ${APACHE_LOG_DIR}/custom.log custom
CustomLog ${APACHE_LOG_DIR}/access.log common
CustomLog ${APACHE_LOG_DIR}/combined.log combined
Although many log management systems support the default Apache logging formats,
it might be best to log in a structured format like JSON since that's the go-to
format for structured logging in the industry and it is universally supported.
Here's a conversion of our custom log format into JSON:
LogFormat "{ \"timestamp\":\"%t\", \"protocol\":\"%H\", \"method\":\"%m\", \"request\":\"%U\", \"query\":\"%q\", \"request_size_in_bytes\":\"%I\", \"status_code\":\"%>s\", \"response_size_in_bytes\":\"%O\", \"time_taken_ms\":\"%{ms}T\" }" json
This produces log entries with the following formatting:
{
"timestamp": "[07/Feb/2022:15:09:02 +0000]",
"protocol": "HTTP/1.1",
"method": "HEAD",
"request": "/index.html",
"query": "?name=john&age=30",
"request_size_in_bytes": "96",
"status_code": "200",
"response_size_in_bytes": "255",
"time_taken_ms": "0"
}
The server error log contains information about any errors that the web server
encountered while processing incoming requests as well as other diagnostic
information. You can choose where the error messages will be transported to
using the ErrorLog directive in your virtual host configuration file. This
transport is usually a log file on the filesystem.
Here is an example from default virtual host configuration file
/etc/apache2/sites-available/000-default.conf:
ErrorLog ${APACHE_LOG_DIR}/error.log
On Debian-based distributions, the default error log is in the
/var/log/apache2/error.log file, while in Fedora/CentOS/RHEL, it placed in the
/var/log/httpd/error_log file. If the path argument to ErrorLog is not
absolute, then it is assumed to be relative to the
ServerRoot.
A common practice is to monitor the error log continuously for any problems
during development or testing. This is easily achieved through the tail
command:
sudo tail -f /var/log/apache2/error.log
You will observe the following output:
[Mon Feb 07 13:03:43.445444 2022] [core:notice] [pid 10469:tid 140561300880448] AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 07 13:07:31.528850 2022] [mpm_event:notice] [pid 10469:tid 140561300880448] AH00491: caught SIGTERM, shutting down
[Mon Feb 07 13:07:31.626878 2022] [mpm_event:notice] [pid 10864:tid 140224997284928] AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations
[Mon Feb 07 13:07:31.626980 2022] [core:notice] [pid 10864:tid 140224997284928] AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 07 13:13:25.966501 2022] [mpm_event:notice] [pid 10864:tid 140224997284928] AH00491: caught SIGTERM, shutting down
[Mon Feb 07 13:13:26.049222 2022] [mpm_event:notice] [pid 11268:tid 139760377875520] AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations
[Mon Feb 07 13:13:26.049318 2022] [core:notice] [pid 11268:tid 139760377875520] AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 07 15:08:50.856388 2022] [mpm_event:notice] [pid 11268:tid 139760377875520] AH00491: caught SIGTERM, shutting down
[Mon Feb 07 15:08:50.940561 2022] [mpm_event:notice] [pid 12096:tid 140473452194880] AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations
[Mon Feb 07 15:08:50.940669 2022] [core:notice] [pid 12096:tid 140473452194880] AH00094: Command line: '/usr/sbin/apache2'
Aside from logging directly to a file, you can also forward your logs to a
Syslog. You can do this by
specifying syslog
instead of a file path as the argument to ErrorLog:
ErrorLog syslog
Like the Apache access logs, the format of the error messages can be controlled
through the ErrorLogFormat directive, which should be placed in the main
config file or virtual host entry. It looks like this:
ErrorLogFormat "[%{u}t] [%l] [pid %P:tid %T] [client\ %a] %M"
The above configuration produces a log entry in the following format:
[Mon Feb 07 15:52:57.234792 2022] [error] [pid 24372:tid 24507] [client 20.113.27.135:34579] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html) found, and server-generated directory index forbidden by Options directive
Here's an explanation of the formatting options used above:
%{u}t: the current time, including microseconds. %l: the log level of the
message. %P: the process identifier. %T: the thread identifier. %a: the
client IP address. %M: the actual log message.
Note that when the data for a formatting option is not available in a particular event, it will be omitted from the log entirely as the Apache error log doesn't use placeholder values for missing parameters.
You can find a complete description of all the available error formatting options in the Apache docs.
In the virtual host configuration file, you can also control the level of
messages that will be entered into the error log through the
LogLevel directive.
When you specify a particular value, messages from all other levels of higher
severity will be logged as well. For example, when LogLevel error is
specified, messages with a severity of crit, alert, and emerg will also be
logged.
LogLevel error
These are the levels available in increasing order of severity:
trace1 - trace8: trace messages (lowest severity).debug: messages used for debugging.info: informational messages.notice: normal but significant conditions.warn: warnings.error: error conditions that doesn't necessarily require immediate action.crit: critical conditions that requires prompt action.alert: errors that require immediate action.emerg: system is unusable.If the LogLevel directive is not set, the server will set the log level to
warn by default.
Storing your Apache logs on the filesystem may suffice for development environments or single-server deployments, but when multiple servers are involved, it may be more convenient to centralize all your logs in a single location so that you can automatically parse, filter, and search log data from all sources in real-time.
In this section, we'll demonstrate how you can centralize your Apache logs in a log management service through Vector, a high-performance tool for building observability pipelines. The following instructions assume that you've signed up for a free Logtail account and retrieved your source token.
Go ahead and follow the relevant installation instructions for Vector for your operating system. On Ubuntu, you may run the following commands to install the Vector CLI:
curl -1sLf \ 'https://repositories.timber.io/public/vector/cfg/setup/bash.deb.sh' \ | sudo -E bash
sudo apt install vector
After Vector is installed, confirm that it is up and running through
systemctl:
systemctl status vector
You should observe that it is active and running:
● vector.service - Vector
Loaded: loaded (/lib/systemd/system/vector.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-02-08 10:52:59 UTC; 48s ago
Docs: https://vector.dev
Process: 18586 ExecStartPre=/usr/bin/vector validate (code=exited, status=0/SUCCESS)
Main PID: 18599 (vector)
Tasks: 3 (limit: 2275)
Memory: 6.8M
CGroup: /system.slice/vector.service
└─18599 /usr/bin/vector
Otherwise, go ahead and start it with the command below.
sudo systemctl start vector
Afterwards, change into a root shell and append your Logtail vector
configuration for Apache into the /etc/vector/vector.toml file using the
command below. Don't forget to replace the <your_logtail_source_token>
placeholder below with your source token.
sudo -s
wget -O ->> /etc/vector/vector.toml \
https://logtail.com/vector-toml/apache2/<your_logtail_source_token>
Then restart the vector service:
sudo systemctl restart vector
You will observe that your Apache logs will start coming through in Logtail:
In this tutorial, you learned about the different types of logs that the Apache web server stores, where you can find those logs, and how to view their contents. We also discussed Apache access and error log formatting and how to create your custom log formats, including a structured JSON format. Finally, we considered how you can manage all your Apache logs in one place by using the Vector CLI to stream each entry to a log management service.
Don't forget to read the docs to find out more about all the logging features that Apache has to offer. Thanks for reading!
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Are you a developer and love writing and sharing your knowledge with the world? Join our guest writing program and get paid for writing amazing technical guides. We'll get them to the right readers that will appreciate them.
Write for us
Write a script, app or project on top of Better Stack and share it with the world. Make a public repository and share it with us at our email.
[email protected]or submit a pull request and help us build better products for everyone.
See the full list of amazing projects on github
