How to Get Started with Logging in Docker

Docker is an open source platform that manages application deployments through abstractions called containers. They are self-contained application environments designed to be portable and reproducible so that the application works reliably and consistently no matter where it is deployed.

When deciding to containerize an application, logging is one of the crucial considerations as its indispensable for troubleshooting, finding trends, and optimizing your application's performance. However, logging in Docker containers presents some unique challenges because containers are ephemeral, so once they are terminated, the logs messages stored in its filesystem are lost. To avoid this, you'll need to transport them to a more permanent location.

In this article, you'll learn the basics of how logging works in Docker containers, how to access and view log messages, and how to craft an optimal logging strategy that suits your application's needs. Note that we'll not be discussing the logs generated by the docker daemon itself. Those logs are helpful for debugging problems with the Docker engine, but the management of such logs is outside the scope of this tutorial.

Prerequisites

Before you proceed with this tutorial, you will need the following:

• A Linux server including a non-root user with sudo access.
• The Docker engine installed (v20.10 or later).
• An understanding of the basics of Docker and containers.

Step 1 — Downloading a sample Docker image

To demonstrate the concepts described in this article, we will use a simple NGINX hello world application image. So, go ahead and use the docker pull command with the image name (karthequian/helloworld) to retrieve download it from the Docker Hub to your server.

docker pull karthequian/helloworld:latest

You'll see the program's output appear on the screen:

Using default tag: latest
latest: Pulling from karthequian/helloworld
83ee3a23efb7: Pull complete
db98fc6f11f0: Pull complete
f611acd52c6c: Pull complete
ce6148ee5b27: Pull complete
f41d580b4c45: Pull complete
272afdecd73d: Pull complete
603e831d3bf2: Pull complete
4b3f00fe862f: Pull complete
1813c5daf2e4: Pull complete
4db7ca47ea28: Pull complete
37d652721feb: Pull complete
e9bce6aacaff: Pull complete
50da342c2533: Pull complete
Digest: sha256:48413fdddeae11e4732896e49b6d82979847955666ed95e4d6e57b433920c9e1
Status: Downloaded newer image for karthequian/helloworld:latest
docker.io/karthequian/helloworld:latest

The output above shows the process of fetching an image and storing it locally to be available for running containers. If you get some permissions error, you may need to prefix the command above with sudo or better still, add the current user to the docker group:

sudo usermod -a -G docker <username>

Once you've done that, log out and log back into your server again, and the docker pull command should work without prefixing it with sudo. You can subsequently use the docker images command to verify that the karthequian/helloworld image is present:

docker images

You should see the following output:

REPOSITORY               TAG       IMAGE ID       CREATED         SIZE
karthequian/helloworld   latest    a0d8db65e6fb   13 months ago   227MB

At this point, you can start the karthequian/helloworld container as an image by executing the docker run command as shown below:

docker run -p 80:80/tcp -d "karthequian/helloworld:latest"

The argument to the -p flag maps port 80 in the container to port 80 on your computer, while the -d option runs the container in the background and prints its ID (typical usage for web service). Assuming there were no errors, the container ID will be displayed on your screen.

39c0ffde9c30600629742357b5f01b278eba9ade7f4c96b9e3883e8fa2b52243

If you get the error below, it means that some other program is already using port 80 on your server, so you should stop that program before rerunning the command.

docker: Error response from daemon: driver failed programming external connectivity on endpoint nostalgic_heisenberg (f493fdf78c94adc66a248ac3fd62e911c1d477dda62398bd36cd40b323605159): Error starting userland proxy: listen tcp4 0.0.0.0:80: bind: address already in use.

You can verify the status of your Docker containers with the docker ps command:

docker ps

You'll see the program's output appear on the screen:

CONTAINER ID   IMAGE                           COMMAND              CREATED          STATUS          PORTS                               NAMES
39c0ffde9c30   karthequian/helloworld:latest   "/runner.sh nginx"   16 seconds ago   Up 15 seconds   0.0.0.0:80->80/tcp, :::80->80/tcp   inspiring_lovelace

The ps command describes a few details about your running containers. You can see the Container ID, the image running inside the container, the command used to start the container, its creation date, the status, exposed ports and the auto-generated name of the container.

If you visit the following URI in your browser http://<your_server_ip> in a browser, you'll see the sample NGINX hello world app.

We will use this web service to demonstrate how logging works in Docker in the following sections.

Step 2 — Viewing Docker container logs

Docker provides a logs command for viewing the log messages produced by a container. When an application in a Docker container emits messages to the standard output and standard error streams, they are aggregated and stored locally on the container host. They will be presented when you specify its ID to docker logs:

docker logs <container_id>

You should see the following output on your screen:

217.138.222.108 - - [21/Feb/2022:12:42:37 +0000] "GET / HTTP/1.1" 200 4369 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
217.138.222.108 - - [21/Feb/2022:12:42:37 +0000] "GET / HTTP/1.1" 200 4369 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
217.138.222.108 - - [21/Feb/2022:12:42:39 +0000] "GET /favicon.ico HTTP/1.1" 404 564 "http://168.119.119.25/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
217.138.222.108 - - [21/Feb/2022:12:42:39 +0000] "GET /favicon.ico HTTP/1.1" 404 564 "http://168.119.119.25/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
217.138.222.108 - - [21/Feb/2022:12:44:25 +0000] "GET / HTTP/1.1" 200 4369 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
217.138.222.108 - - [21/Feb/2022:12:44:25 +0000] "GET / HTTP/1.1" 200 4369 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"

Note that if your application does not log to the stdout or stderr, you may not see any useful information when running the docker logs command.

The docker logs command displays only the log entries present at the time of execution. If you want to continue streaming output from the containers stdout and stderr, use the --follow option:

docker logs --follow <container_id>

You can also use the --tail flag to display a specific number of lines from the end of the logs:

docker logs --tail 10 <container_id> # show the last 10 lines

Two other valuable flags are --since and --until. The former specifies the lower time limit, while the latter sets an upper time limit for which logs should be displayed. For example, the command below will only display log entries that were produced between 12PM and 1PM on Feb 21, 2022 on the specified container.

docker logs --since 2022-02-21T12:00:00 --until 2022-02-21T13:00:00 <container_id>

You can also filter the contents of docker logs through grep if you're only searching for lines containing a specific pattern:

docker logs <container_id> | grep <pattern>

Step 3 — Working with Docker logging drivers

The mechanism used by Docker to retrieve information from a Docker container is known as a logging driver. The Docker daemon uses a default logging driver for all containers unless a different one is specified. This default logging driver is usually the json-file driver, and it is used to format container logs as JSON objects and cache them in plain text files.

Below are some of the logging drivers that Docker supports:

• none: no logs are available for the container.
• local: logs are stored in a custom format designed for minimal overhead.
• json-file: the logs are formatted as JSON and stored in the container.
• syslog: writes logging messages to the syslog facility on the host machine. You can read our tutorial on syslog if you are not familiar with it.
• journald: writes log messages to the journald utility on the host machine. See our tutorial on journald for more details.
• fluentd: writes log messages to the Fluentd. The fluentd daemon must be running on the host machine.
• gelf: sends log entries to a Graylog Extended Log Format (GELF) endpoint such as Graylog or Logstash.
• awslogs: sends log entries to AWS CloudWatch.
• gcplogs: sends log entries to the Google Cloud Platform.
• logentries: sends log entries to Rapid7 Logentries.
• splunk: uses the HTTP Event Collector to write log messages to Splunk.
• etwlogs (Windows only): writes log entries as Event Tracing for Windows (ETW) events.

You can check the current logging driver for the docker daemon through the command below:

docker info --format '{{.LoggingDriver}}'

This should yield the output below:

json-file

Step 4 — Configuring the json-file logging driver

In this section, we will configure the json-file driver for our running Docker container for demonstration purposes. Although this is already the default driver, log rotation and compression are not performed by default so we'll ensure that both options are configured so that log files are kept to manageable sizes.

Start by opening or creating the daemon configuration file in /etc/docker/daemon.json:

sudo nano /etc/docker/daemon.json

Populate the file with the following contents:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

The log-driver sets the logging driver to json-file while the log-opt property sets the supported options for the specified logging driver. Note that each property in the log-opts object must have a string value (including boolean and numeric values as above).

Here's a brief explanation of each option used above:

• max-size: the maximum size of the log file before it is rotated. An integer plus a modifier represents the measuring unit (k - kilobytes, m - megabytes, or g - gigabytes). Defaults to -1 (unlimited).
• max-file: the maximum number of rotating logs. The max-size must be specified.
• compress: enable compression for rotated logs.

The complete list of all the options for the json-file driver is defined in the official documentation.

Save the configuration file and restart the docker service to apply the changes to newly created containers. Existing containers will not adopt the new configuration unless they are re-run.

sudo systemctl restart docker

After restarting the docker service, all your running containers will be terminated, so you must start them again with docker run:

docker run -p 80:80/tcp -d "karthequian/helloworld:latest"

You can use the live-restore property in your daemon configuration file to ensure that containers continue running even if the daemon becomes unavailable.

{
  "live-restore": true,
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

If you want to override the default logging driver or one of the logging options for a specific container, you can use the --log-driver and --log-opt flags when starting a container as shown below:

docker run --log-driver syslog --log-opt syslog-address=udp://1.2.3.4:1111 -p 80:80/tcp -d "karthequian/helloworld:latest"

Step 5 — Viewing the json-file log entries

The json-file logging driver configured in the previous step stores the logs on the host server, and you can examine them in your text editor or through the cat utility. As described in step 2, you can view the logs for a specific container through the docker logs command. If you want to examine the actual log file, you can do so by locating the following file:

/var/lib/docker/containers/<container_id>/<container_id>-json.log

Note that <container_id> above represents the full container id not the shorter one what is usually presented. You can get the path to the log file for a specific container through the command below. Use docker inspect to verify the ID of your running container (if you don't know it already).

docker inspect -f {{.LogPath}} <container_id>

You'll observe the following output on your screen:

/var/lib/docker/containers/ec539cf34cb8901f8504de6d28e40f16c6f4bb47f33c37298c37ce61b308cd58/ec539cf34cb8901f8504de6d28e40f16c6f4bb47f33c37298c37ce61b308cd58-json.log

Go ahead and examine the contents of this file with cat:

cat /var/lib/docker/containers/<container_id>/<container_id>-json.log

You'll observe the following output:

{"log":"217.138.222.108 - - [22/Feb/2022:10:38:06 +0000] \"GET / HTTP/1.1\" 200 4369 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36\"\n","stream":"stdout","time":"2022-02-22T10:38:06.354603159Z"}
{"log":"217.138.222.108 - - [22/Feb/2022:10:38:06 +0000] \"GET / HTTP/1.1\" 200 4369 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36\"\n","stream":"stdout","time":"2022-02-22T10:38:06.354645627Z"}

Each log entry has the following three properties:

• log: the log message.
• stream: the origin of the log entry (stdout, or stderr).
• time: a timestamp that indicates when the log was recorded.

The docker logs command displays only the contents of the log property, but you can also show the time property by using the --timestamps option.

docker logs --timestamps <container_id>

2022-02-22T10:38:06.354645627Z 217.138.222.108 - - [22/Feb/2022:10:38:06 +0000] "GET / HTTP/1.1" 200 4369 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
2022-02-22T11:07:03.852689039Z 172.105.189.111 - - [22/Feb/2022:11:07:03 +0000] "GET / HTTP/1.1" 200 4369 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
2022-02-22T11:07:03.852988169Z 172.105.189.111 - - [22/Feb/2022:11:07:03 +0000] "GET / HTTP/1.1" 200 4369 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"

Prior to Docker 20.10, the docker logs command only worked when the logging driver was set to local, json-file, and journald, but this changed with the introduction of dual logging in Docker 20.10. This feature allows docker logs to read container logs locally in a consistent format regardless configured logging driver that is being used. It's enabled by default, but if you prefer to disable it, use the cache-disabled property shown below.

{
  "log-driver": "splunk",
  "log-opts": {
    "cache-disabled": "true"
  }
}

Note that the cache-disabled option does not affect the local, json-file, or journald drivers since they do not use the dual logging feature.

Step 6 — Choosing a delivery mode from container to log driver

The logging delivery mode for a Docker container refers to how it prioritizes the delivery of incoming log messages to the driver. The following two modes are supported, and they can be used with any logging driver:

1. Blocking mode

In blocking mode (the default), the delivery of log messages to the driver will block all other operations that the container is performing, which may impact its performance, especially with drivers that write to a remote service. The main advantage of blocking mode is that it guarantees that each log message will be delivered to the driver.

{
  "log-driver": "json-file",
  "log-opts": {
    "mode": "blocking"
  }
}

You can keep using the blocking mode when the logging driver being used writes to the local filesystem. It is unlikely that the latency introduced will be significant since these drivers are generally rapid.

2. Non-blocking mode

In non-blocking mode, incoming log entries are stored in a memory buffer until the configured logging driver is available to process them. Once the logs are processed, they are cleared from the buffer to make way for new entries.

{
  "log-driver": "syslog",
  "log-opts": {
    "mode": "non-blocking"
  }
}

The advantage of this mode is that the application's performance will not be impacted. However, it also introduces the possibility of losing log entries if the memory buffer is filled up so that existing entries are deleted before processing. To decrease the likelihood of losing log messages in containers that generate a significant amount of logs, you can increase the maximum memory buffer size from its default (1MB) through the max-buffer-size property:

{
  "log-driver": "syslog",
  "log-opts": {
    "mode": "non-blocking",
    "max-buffer-size": "5m"
  }
}

Step 7 — Choosing a logging strategy

There are several ways to aggregate, store and centralize your Docker container logs. Thus far, we've only covered the native Docker logging driver approach in this article since it's the most suitable for many everyday use cases. However, that strategy won't fit every project, so it's a good idea to be aware of alternative methods so that you can make an informed decision about the right logging approach for your application.

1. Using application-based logging

In this approach, the application itself handles its own logging through a logging framework. For example, a Node.js application could use Winston or Pino to format and transport its logs to a log management solution for storage and further processing.

This approach provides the greatest amount of control for application developers to generate, format, and transport the logs as they see fit. However, this control has a performance cost since everything (including the complexities of log delivery) is done at the application level.

Another consideration is that logs must be transported to a remote log management service (such as Logtail) or a data volume outside the container's filesystem to prevent loss on container termination.

2. Using a logging driver

Instead of using the application's logging framework to transport your logs, you can log to the stdout and stderr streams, then use a logging driver to manage how they are stored or transported. This is the native way to log in Docker, and it reduces the impact of logging on your application's performance. The main downside to this approach is that it creates a dependency between your container and its host, but this is tolerable in most situations.

3. Using data volumes

If you don't want your container logs to be lost immediately after the container is terminated, you can link a directory inside the container to a directory on the Docker host where the log entries are transported to. This ensures that your logs are retained even when the container is destroyed, and it also makes it easy to aggregate logs from multiple containers in one place and make copies or transport them to some remote location.

4. Using a dedicated logging container

Setting up a dedicated container whose sole purpose is to aggregate and centralize Docker logs could be a great solution, especially when deploying a microservice architecture. This approach removes the dependency on the host machine and makes it easier to scale up your logging infrastructure by simply adding a new container when needed.

5. Using the sidecar approach

A technique used for more complicated deployments is the sidecar approach in which each Docker container has its own dedicated logging container (they are considered a single unit). The main advantage is that the logging approach can be tailored for each application, and it offers greater transparency regarding the origin of each log entry.

A drawback to this strategy is that setting up a logging container per application will consume more resources than the dedicated logging container approach, and the added complexity may not be worthwhile for smaller deployments. You also need to use Docker compose to ensure that both containers are managed as a single unit to avoid incomplete or missing log data.

Conclusion

This tutorial provided an introduction to logging in Docker containers that should help you get set up quickly when deploying your applications to production in a Docker container. We started by examining how logging works in Docker containers, where the logs are stored, and how to view them. Afterward, we examined the concept of logging drivers in Docker and how to configure them. Then, we rounded off the article by discussing alternative approaches to consider for an optimal logging set up in Docker.

If you'd like to learn more about logging in Docker containers, feel free to explore other articles in this series, or check out the official documentation on the subject.

Thanks for reading, and happy logging!

Article by

Ayooluwa Isaiah

Ayo is a technical content manager at Better Stack. His passion is simplifying and communicating complex technical ideas effectively. His work was featured on several esteemed publications including LWN.net, Digital Ocean, and CSS-Tricks. When he's not writing or coding, he loves to travel, bike, and play tennis.

Got an article suggestion? Let us know

Next article

How to Start Logging With Heroku

Learn how to start logging with Heroku and go from basics to best practices in no time.

→

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.