How to Get Started with Logging in Go

Logging is helpful for more than just tracking error conditions in your application. It's also a great way to record any notable events that occur during the lifetime of a program so that you can have a good idea of what is going on and what needs to be changed, dropped, or optimized further. Adopting good logging practices provides valuable insights into the flow of your program, and what parameters are responsible for various events, which is immensely valuable when trying to reproduce problems in your application.

In this tutorial, we will discuss the basics of logging in Go starting with the built-in log package, and then proceed to discuss third-party logging frameworks and how they go far beyond what the default log package has to offer. By following through with this article, you'll learn the following Go logging concepts:

Prerequisites

Before proceeding with this article, ensure that you have a recent version of Go installed on your computer so that you can run the snippets and experiment with some of the concepts that will be introduced in the following sections.

When to log

Logs need to communicate the various happenings in your application effectively so they must be descriptive and provide enough context for you to understand what happened, and what (if anything) needs to be done next. Logs are also reactive because they can only tell you about something that has already happened in your application.

You can read the log and take some action afterward to prevent that event from happening again, but if you don't have the log in the first place, you will be at a disadvantage when trying to find the cause of a problem or gain insight into some notable events.

Therefore, logging should start as early as possible during the development process and it should be kept in place when the program is deployed to production. Here are some general recommendations for where to log in your application:

• At the beginning of an operation (e.g at the start of a scheduled job or when handing incoming HTTP requests).
• When a significant event occurs while the operation is in progress, especially if the flow of the code changes entirely due to such an event.
• When an operation is terminated regardless of whether it succeeds or fails.

What should you log

You should log all notable events in your program, and add sufficient context to help you understand what triggered the event. For example, if a server error occurs, the reason for the error and a stack trace should be included in the corresponding log entry. If a failed login attempt is detected, you can log the IP address of the client, user agent, username or id, and the reason for failure (password incorrect, username invalid, etc.).

A high number of failed logins attempts for non-existent users or quickly reaching the login attempts limits for many accounts may be an indicator of a coordinated attack on your service, and you can set up alerts to draw your attention to such issues when they arise if you're processing your logs through a log management service (see example from Logtail below.

Let's look at a practical example of what to include in a log entry for each incoming HTTP request to your server. You should log at least the following:

• the route and query parameters (if any),
• request method (GET, POST, etc),
• response code,
• user agent, and
• time taken to complete the request.

Armed with this data, you can figure out your busiest routes, average response times, if your service is experiencing some degradation in performance (due to increased server errors or response times), and much more. You can also use the captured data to build graphs and other visualizations, which could come in handy when discussing trends and opportunities for future investment with the management at your organization.

In the case of unexpected server errors (5xx), it may also be necessary to log the request headers and body so that you can reproduce the request and track down the issue, but ensure that you are careful not to leak sensitive details when doing so as the headers and bodies of certain requests can contain sensitive info like passwords, API keys, or credit card information. If you do decide to log such data, ensure to write specific rules to sanitize it and redact all the sensitive fields from the log entry.

What to avoid when logging

Generally, you should include as many details as possible in your log entries, but you must also be careful not to add details that will make your logs unnecessarily verbose as this will impact storage requirements and potentially increase the cost of managing your logs.

As alluded to in the previous section, you should also avoid logging sensitive information or Personally Identifiable Information (PII) such as phone numbers, home addresses, and similar details, so you don't fall afoul of regulations like GDPR, CCPA, and other data compliance laws.

The Go standard library log package

Now that we've discussed a general strategy to help you get started quickly with logging, let's discuss the how of logging in Go applications. In this section, We'll explore the built-in log package in the standard library designed to handle simple logging concerns. While this package does not meet our criteria for a good logging framework due to some missing features, it's still necessary to be familiar with how it works as many in the Go community rely on it.

Here's the most basic way to write a log message in Go:

package main

import "log"

func main() {
    log.Println("Hello from Go application!")
}

2022/08/10 11:19:52 Hello from Go application!

The output contains the log message and a timestamp in the local time zone that indicates when the entry was generated. Println() is one of methods accessible on the preconfigured logger prints its output to the standard error. The following other methods are available:

log.Print()
log.Printf()
log.Fatal()
log.Fatalf()
log.Fatalln()
log.Panic()
log.Panicf()
log.Panicln()

The difference between the Fatal and Panic methods above is that the former calls os.Exit(1) after logging a message, while the latter calls panic().

log.Fatalln("cannot connect to the database")

2022/08/10 14:32:51 cannot connect to database
exit status 1

log.Panicln("cannot connect to the database")

2022/08/10 14:34:07 cannot connect to database
panic: cannot connect to database

goroutine 1 [running]:
log.Panicln({0xc00006cf60?, 0x0?, 0x0?})
        /usr/local/go/src/log/log.go:402 +0x65
main.main()
        /home/ayo/dev/demo/random/main.go:6 +0x45
exit status 2

If you want to customize the default logger, you can call log.Default() to access it and then call the appropriate methods on the returned Logger object. For example, you can change the output of the logger to stdout as shown below:

package main

import (
    "log"
    "os"
)

func main() {
    defaultLogger := log.Default()
    defaultLogger.SetOutput(os.Stdout)
    log.Println("Hello from Go application!")
}

You can also create a completely custom logger through the log.New() method which has the following signature:

func New(out io.Writer, prefix string, flag int) *Logger

The first argument is the destination of the log messages produced by the Logger, which can be anything that implements the io.Writer interface. The second is a prefix that is prepended to each log message, while the third specifies a set of constants that is used to add details to each log message.

package main

import (
    "log"
    "os"
)

func main() {
    logger := log.New(os.Stderr, "", log.Ldate|log.Ltime)
    logger.Println("Hello from Go application!")
}

The above logger is configured to output to the standard error, and it uses the initial values for the standard logger, which means the output from the logger is the same as before:

2022/08/10 11:19:52 Hello from Go application!

We can customize it further by adding the application name, file name, and line number to the log entry. We'll also add microseconds to the timestamp and cause it to be presented in UTC instead of the local time zone:

logger := log.New(
  os.Stderr,
  "MyApplication: ",
  log.Ldate|log.Ltime|log.Lmicroseconds|log.LUTC|log.Lshortfile,
)
logger.Println("Hello from Go application!")

The output becomes:

MyApplication: 2022/08/10 13:55:48.380189 main.go:14: Hello from Go application!

The MyApplication: prefix appears at the beginning of each log entry and the timestamp (now UTC instead of the local time) now includes microseconds. The point of log generation also included in the output to help you locate the source of each entry in the codebase.

Logging to a file in Go

So far, we've seen several examples that log to the standard output or standard error. Let's now address the common need to transport logs into a file and also how to rotate such log files to prevent them from growing too large which can make them cumbersome to work with.

Using shell redirection

The easiest way to output logs into a file is to keep logging to the console and then use shell redirection to append each entry to a file:

go run main.go 2>> myapp.log

The above command redirects the standard error stream to a myapp.log file in the current directory.

cat myapp.log

MyApplication: 2022/08/29 10:05:25.477612 main.go:14: Hello from Go application!

If you're logging to the standard output, you can use 1>> or >> instead:

go run main.go >> myapp.log

To redirect both the standard output and standard error streams to a single file, you can use &>>.

go run main.go &>> myapp.log

You can also redirect each output stream to separate files as shown below:

go run main.go 2>> stderr.log >> stdout.log

Finally, you can use the tee command to retain the console output while redirecting one or both streams to a file:

go run main.go 2> >(tee -a stderr.log >&2)

go run main.go > >(tee -a stdout.log)

go run main.go > >(tee -a stdout.log) 2> >(tee -a stderr.log >&2)

Writing log messages directly to a file

A second way to log into files in Go is to open the desired file in the program and write to it directly. When using this approach, you need to ensure that the file is created or opened with the right permissions to ensure that the program can write to it.

package main

import (
    "log"
    "os"
)

func main() {
    // create the file if it does not exist, otherwise append to it
    file, err := os.OpenFile(
        "myapp.log",
        os.O_APPEND|os.O_CREATE|os.O_WRONLY,
        0664,
    )
    if err != nil {
        panic(err)
    }

    defer file.Close()

    logger := log.New(
        file,
        "MyApplication: ",
        log.Ldate|log.Ltime|log.Lmicroseconds|log.LUTC|log.Lshortfile,
    )
    logger.Println("Hello from Go application!")
}

Since the os.File type implements a Write() method that satisfies the io.Writer interface, we can open any file and use it as the output for our logger as shown above. If you execute the code, you'll notice that the log message is placed in a myapp.log file in the current directory:

cat myapp.log

. . .
MyApplication: 2022/08/10 14:12:24.638976 main.go:26: Hello from Go application!

You can also use the io.MultiWriter() method to log to multiple writers at once such as multiple files or a file and the standard error (or output) at once:

. . .
logger := log.New(
  io.MultiWriter(file, os.Stderr), // log to both a file and the standard error
  "MyApplication: ",
  log.Ldate|log.Ltime|log.Lmicroseconds|log.LUTC|log.Lshortfile,
)
. . .

When you run the program again, you'll notice that the log entry is recorded to the console and the myapp.log file simultaneously.

Rotating log files

When logging to files, you should take the time to setup a log rotation policy so that the file sizes are kept manageable, and older logs get deleted automatically to save storage space.

While you can rotate the files yourself by appending a timestamp to the log filename coupled with some checks to delete older files, we generally recommend using an external program like logrotate, a standard utility on Linux , or a well tested third-party Go package for this purpose.

We prefer the former approach since it's the standard way to solve this problem on Linux, and it is more flexible than other solutions. It can also copy and truncate a file so that file deletion doesn't occur in the middle of writing a log entry which can be disruptive to the application. If you want to learn more about rotating log files using logrotate, please see the linked tutorial on the subject.

You can also utilize Lumberjack a rolling file logger package for Go applications as shown below:

package main

import (
    "log"

    "gopkg.in/natefinch/lumberjack.v2"
)

func main() {
    fileLogger := &lumberjack.Logger{
        Filename:   "myapp.log",
        MaxSize:    10, // megabytes
        MaxBackups: 10, // files
        MaxAge:     14,   // days
        Compress:   true, // disabled by default
    }

    logger := log.New(
        fileLogger,
        "MyApplication: ",
        log.Ldate|log.Ltime|log.Lmicroseconds|log.LUTC|log.Lshortfile,
    )
    logger.Println("Hello from Go application!")
}

With the above configuration in place, the myapp.log file will continue to be populated until it reaches 10 megabytes. Afterward, it will be renamed to myapp-<timestamp>.log (such as myapp-2022-08-10T18-30-00.000.log) and a new file with the original name (myapp.log) is created. The backup files are also gzip compressed so that they take up less space on the server, but you can disable this feature if you want. When the number of backup files exceeds 10, the older ones will be deleted automatically. The same thing happens if the file exceeds the number of days configured in MaxAge regardless of the number of files present.

Limitations of the log package

While the log package is a handy way to get started with logging in Go, it does have several limitations that make it less than ideal for production applications whose logs are meant to be processed by machines for monitoring and further analysis.

Lack of log levels

[Levelled logging][log-levels] are one of the most sought-after features in a logging package, but they are strangely missing from the log package in Go. You can fix this omission by create a custom log package that builds on the standard log as demonstrated in this GitHub gist. It provides the ability to use leveled methods (Debug(), Error(), etc), and turn off certain logs based on their level.

You can utilize the custom log package in your application as follows:

package main

import (
    "log"
    "os"

    "github.com/username/project/internal/logger"
)

func main() {
    l := log.New(
        os.Stderr,
        "MyApplication: ",
        log.Ldate|log.Ltime|log.Lmicroseconds|log.LUTC,
    )

    logger.SetLevel(2) // default to the info level
    logger.SetLogger(l) // Override the default logger

    logger.Trace("trace message")
    logger.Debug("debug message")
    logger.Info("info message")
    logger.Warn("warning message")
    logger.Error("error message")
    logger.Fatal("fatal message")
}

Notice that the Trace and Debug() methods do not produce any output proving that the call to logger.SetLevel() had the desired effect:

MyApplication: 2022/08/10 16:22:45.146744 [INFO] info message
MyApplication: 2022/08/10 16:22:45.146775 [WARN] warning message
MyApplication: 2022/08/10 16:22:45.146789 [ERR] error message
MyApplication: 2022/08/10 16:22:45.146800 [FATAL] fatal message

2. Lack of support for structured logging

The use of basic or unstructured logs is being gradually phased out in the tech industry as more organizations adopt the use of a structured format (usually JSON) which enables each log entry to be treated as data that can be automatically parsed by machines for monitoring, alerting, and other forms of analysis.

Due to this trend, many logging frameworks have added structured logging APIs to their public interface, and some support structured logging only. However, the log package in Go is not of them as it only supports the basic string formatted logs through its printf-style methods. If you desire to output your Go logs in a structured format, you have no choice but to use a third-party logging framework. In the next section, we will consider a few packages that make structured logging in Go a breeze.

Third-party logging libraries to consider

Due to the above limitations, the standard library log package in Go should generally be used in logging contexts where humans are the primary audience of the logs. For everyone else, it's necessary to adopt one of the Go logging packages discussed below:

1. Zerolog

Zerolog is a structured logging package for Go that boasts of a great development experience and impressive performance when compared to alternative libraries. It offers a chaining API that allows you to specify the type of each field added to a log entry which helps with avoiding unnecessary allocations and reflection. Zerolog only supports JSON or the lesser-known Concise Binary Object Representation (CBOR) format, but it also provides a way to prettify its output in development environments.

package main

import (
    "github.com/rs/zerolog/log"
)

func main() {
    log.Info().
        Str("name", "John").
        Int("age", 9).
        Msg("hello from zerolog")
}

{"level":"info","name":"John","age":9,"time":"2022-08-11T21:28:12+01:00","message":"hello from zerolog"}

You can import a pre-configured global logger (as shown above) or use zerolog.New() to create a customizable logger instance. You can also create child loggers with additional context which can come in handy for logging in various packages or components in an application. Zerolog also helps you adequately log errors by providing the ability to include a formatted stacktrace through its integration with the popular errors package. It also provides a set of helper functions for better integration with HTTP handlers.

2. Zap

Uber's Zap library pioneered the reflection-free, zero-allocation logging approach adopted by Zerolog. Still, it also supports a more loosely typed API that can be used when ergonomics and flexibility are the overriding concern when logging. This less verbose API (zap.SugaredLogger) supports both structured and formatted string logs, while the base zap.Logger type supports only structured logs in JSON format by default. The good news is that you don't have to pick one or the other throughout your codebase. You can use both and convert between the two freely at any time.

package main

import (
    "fmt"
    "time"

    "go.uber.org/zap"
)

func main() {
    // returns zap.Logger, a strongly typed logging API
    logger, _ := zap.NewProduction()

    start := time.Now()

    logger.Info("Hello from zap Logger",
        zap.String("name", "John"),
        zap.Int("age", 9),
        zap.String("email", "[email protected]"),
    )

    // convert zap.Logger to zap.SugaredLogger for a more flexible and loose API
    // that's still faster than most other structured logging implementations
    sugar := logger.Sugar()
    sugar.Warnf("something bad is about to happen")
    sugar.Errorw("something bad happened",
        "error", fmt.Errorf("oh no!"),
        "answer", 42,
    )

    // you can freely convert back to the base `zap.Logger` type at the boundaries
    // of performance-sensitive operations.
    logger = sugar.Desugar()
    logger.Warn("the operation took longer than expected",
        zap.Int64("time_taken_ms", time.Since(start).Milliseconds()),
    )
}

{"level":"info","ts":1660252436.0265622,"caller":"random/main.go:16","msg":"Hello from zap Logger","name":"John","age":9,"email":"[email protected]"}
{"level":"warn","ts":1660252436.0271666,"caller":"random/main.go:24","msg":"something bad is about to happen"}
{"level":"error","ts":1660252436.0275867,"caller":"random/main.go:25","msg":"something bad happened","error":"oh no!","answer":42,"stacktrace":"main.main\n\t/home/ayo/dev/demo/random/main.go:25\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
{"level":"warn","ts":1660252436.0280342,"caller":"random/main.go:33","msg":"the operation took longer than expected","time_taken_ms":1}

Unlike Zerolog, Zap does not provide a functioning global logger by default but you can configure one yourself through its ReplaceGlobals() function. Another difference between the two is that Zap does not support the TRACE log level at the time of writing, which may be a deal-breaker for some. In Zap's favor, you can greatly customize its behavior by implementing the interfaces in the Zapcore package. For example, you can output your logs in a custom format (like lgofmt), or transport them directly to a log aggregation and monitoring service like Logtail.

Honourable mentions

Logrus was the default structured logging framework for Go for a long time until it was surpassed by the aforementioned Zap and Zerolog. Its major advantage was that it was API compatible with the standard library log package while providing structured, leveled logging in JSON or other formats. It is currently in maintenance mode so no new features will be added, but it will continue to be maintained for security, bug fixes, and performance improvements where possible.

Log15's primary goal is to provide a structured logging API that outputs logs in a human and machine readable format. It uses the Logfmt format by default to aid this goal although this can easily be changed to JSON. It also provides built-in support for logging to files, Syslog, and the network, and it is also quite extensible through its Handler interface.

Apex/log is a structured logging framework inspired by Logrus, but with a simplified API along with several built-in handlers that help to facilitate log centralization. At the time of writing, it has not been updated in two years so we're not sure if it's still being maintained.

Logr is not a logging framework, but an interface that aims to decouple the practice of structured logging in a Go application from a particular implementation. This means you get to write all your logging calls in terms of the APIs provided on the logr.Logger interface, while the actual logging implementation (Zap, Zerolog, or something else) is managed in one place to ease future migrations.

Final thoughts

The log package in the Go standard library provides a simple way to get started with logging, but you will likely find that you need to reach out for a third-party framework to solve many common logging concerns. Once you've settled on a solution that works for your use case, you should consider supercharging your logs by sending them to a log management platform, where they can be monitored and analyzed in-depth.

Thanks for reading, and happy logging!

Article by

Ayooluwa Isaiah

Ayo is a technical content manager at Better Stack. His passion is simplifying and communicating complex technical ideas effectively. His work was featured on several esteemed publications including LWN.net, Digital Ocean, and CSS-Tricks. When he's not writing or coding, he loves to travel, bike, and play tennis.

Got an article suggestion? Let us know

Next article

A Complete Guide to Logging in Go with Zerolog

Zerolog is a high-performance Go structured logging library aimed at latency-sensitive applications where garbage collections are undesirable

→

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.