How to Start Logging With Heroku

Heroku is a platform that lets you deploy, run and manage applications written in various programming languages like Python, Java, C#, JavaScript, PHP and others. The purpose of Heroku is to give you the freedom to focus on the applications rather than the administration of the infrastructure. The infrastructure administrations typically includes logging. Heroku offers a high-level tool for log maintenance.

In this tutorial, you will do following actions with the Heroku platform in general:

• You will learn the concept of Heroku logging and what it offers, which type of logs are maintained, and what is the log record structure.
• You will view the Heroku logs through the command-line interface. You will filter the logs by the source.

Next, you will create the application log for your service in the platform-specific languages (Python, Java, C#, JavaScript, and PHP) and configure it in such a way that it will be maintained by Heroku:

• You will view and write the application log for the Python application in the Flask web framework.
• You will use the Java facade SLF4J with the logging framework Log4J, and you will configure the log record structure in the independent XML configuration file.
• You will integrate the C# logging framework Log4Net into your .NET application. You will configure the log record structure in the XML configuration file.
• You will write REST API logging for the Node.js application written in the Express.js framework. You will include the REST API logging framework Morgan into the Express.js application.
• You will set up logging for your PHP application and you will log your application written in the Laravel application framework.

Prerequisites

The tutorial expects that you already install the Heroku and integrate it into your application. You should be familiar with the Heroku dynos, and add-ons.

• The first part of the tutorial is platform-independent. We expect that you already install the Heroku and deploy your application to the remote git repository. We will use the repository https://your-domain.herokuapp.com/ for demonstration purpose.
• The second part of the tutorial shows examples of how to start logging with Heroku for the different application languages. You can follow only the step with the language you are interested in.

Step 1 — Understanding the Heroku Logging

Heroku logging offers a more abstract layer over the different application logs. The application consists of multiple components such as database, web routing with REST API and others. Heroku could gather the logs from the different sources and use some Heroku add-on for their visualization.

The heroku maintains following type of source for the log:

• Application logs: Log records generated in the application code. The format of the messages is in the hands of application developers.
• System logs: These log records generates the Heroku platform. It reports actions such as building, deploys, or process crashes.
• API logs: The messages report the application administrative actions executed by the developers (deploy new code, scale the dyno resources, rollbacks).
• Add-on logs: Log records generated by the add-on services. You can view the full list of the supported add-ons with their documentation for the details about their logging.

Structure of the Log

Each log record has following four fields:

timestamp source[dyno]: message

• Timestamp: The timestamps have accurately generated when the dyno generates the event. The timestamp has a standardized format RFC5424. The default timezone is UTC (Universal Time Zone).
• Source: The source that generates the log record. It could be the application itself (source is labelled as the app), the Heroku platform (the source is heroku), some third-party add-on, and many others.
• Dyno: The name of process (dyno in heroku terminology) that generates the log record.
• Message: String with a raw log message. The message includes a human readable record about the event, but also furthermore metadata (for example, inner timestamp, or severity) with a format suitable for the type of the event (for example, heroku URI routing includes fields from HTTP request).

The real log record could looks like:

2021-06-01T08:25:21.297420+00:00 app[api]: Initial release by user [email protected]

As you can see, the example includes all four log record fields.

Step 2 — Viewing Logs with Heroku CLI

Heroku includes the CLI (command-line interface) that also includes the log manipulation commands. It allows to view and filter the log records. You can view recent log output by executing heroku with option logs:

$ heroku logs

You'll see the program's output appear on the screen:

Output
2021-06-01T08:25:21.297420+00:00 app[api]: Initial release by user [email protected]
2021-06-01T08:25:21.297420+00:00 app[api]: Release v1 created by user [email protected]
2021-06-01T08:25:21.553252+00:00 app[api]: Release v2 created by user [email protected]
2021-06-01T08:25:21.553252+00:00 app[api]: Enable Logplex by user [email protected]
2021-06-01T08:26:22.055146+00:00 heroku[router]: at=info code=H81 desc="Blank app" method=GET path="/" host=your-domain.herokuapp.com request_id=84008b3c-68ef-4a69-a452-f200ed5c2f7d fwd="95.103.134.130" dyno= connect= service= status=502 bytes= protocol=https
2021-06-01T08:26:22.339664+00:00 heroku[router]: at=info code=H81 desc="Blank app" method=GET path="/favicon.ico" host=your-domain.herokuapp.com request_id=ba4cedfa-82b1-49eb-a190-dc4f4841bd13 fwd="95.103.134.130" dyno= connect= service= status=502 bytes= protocol=https
...

The output shows 100 lines where each line holds a single log record. As you can see, the log record structure is as described in the previous step.

Viewing Specific Number of Log Lines

If you want to view more (or fewer) lines then you can re-execute the same command with parameter -n:

$ heroku logs -n 3

The parameter -n 3 determines to display exactly 3 lines. You'll see the program's output appear on the screen:

Output
2021-06-01T08:25:21.297420+00:00 app[api]: Initial release by user [email protected]
2021-06-01T08:25:21.297420+00:00 app[api]: Release v1 created by user [email protected]
2021-06-01T08:25:21.553252+00:00 app[api]: Release v2 created by user [email protected]

The output shows exactly three log records. However, you can display up to 1500 lines.

Viewing Log Tail in Real-time

If you want to view your latest log records in real-time for the live application debugging purpose, then you can use the --tail option, which displays the tail of recent logs in real-time. Let's execute the heroku logs with the --tail option:

$ heroku logs --tail

Now, you can reload a few times your Heroku domain https://your-domain.herokuapp.com/ in your browser to generate some log records. The output will generate the new log records in real-time (this works only if your application logs such an event as an HTTP request, we will show how to configure such an application logger in the next steps):

Output
...
2021-06-01T12:32:10.289571+00:00 heroku[router]: at=info method=GET path="/" host=your-domain.herokuapp.com request_id=c387dfb7-ec98-4591-8abb-6f68d98e500a fwd="95.103.134.130" dyno=web.1 connect=0ms service=2ms status=200 bytes=165 protocol=https
2021-06-01T12:32:10.288845+00:00 app[web.1]: 10.5.201.229 - - [01/Jun/2021:12:32:10 +0000] "GET / HTTP/1.1" 200 12 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
2021-06-01T12:32:11.590971+00:00 heroku[router]: at=info method=GET path="/" host=your-domain.herokuapp.com request_id=91855b5a-ab36-4e0b-aa5d-0d7089c08794 fwd="95.103.134.130" dyno=web.1 connect=0ms service=2ms status=200 bytes=165 protocol=https
2021-06-01T12:32:11.590438+00:00 app[web.1]: 10.5.201.229 - - [01/Jun/2021:12:32:11 +0000] "GET / HTTP/1.1" 200 12 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
2021-06-01T12:32:13.064383+00:00 heroku[router]: at=info method=GET path="/" host=your-domain.herokuapp.com request_id=6f16762f-38a2-48ec-b801-4e63b6dcccc9 fwd="95.103.134.130" dyno=web.1 connect=0ms service=1ms status=200 bytes=165 protocol=https
2021-06-01T12:32:13.064135+00:00 app[web.1]: 10.5.201.229 - - [01/Jun/2021:12:32:13 +0000] "GET / HTTP/1.1" 200 12 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"

The end of output should display the log records similar to these. You can see that the log records reports about the HTTP request to the specified URI.

Filtering the Logs

As described in the previews step, each log records refers to a specific source and dyno. You can filter the log by these fields.

If you want to view only the application logs (logs from the application itself) then execute the heroku logs with the option --source app:

$ heroku logs --source app

In case that you want to show the system logs (log records about actions taken by the Heroku platform) then re-execute the same command, but with --source heroku:

$ heroku logs --source heroku

If you are looking for the API logs (messages about administrative actions) then specify also --dyno option to the api:

$ heroku logs --source app --dyno api

These are some examples of possible sources. If you use any add-on then you can filter the log records generated by them.

The log records displayed with the Heroku CLI are not in chronological**order**. These log records are generated by various sources into a single log stream, thus the parallel processes can write to the log in unordered way.

Step 3 — Python Application Logging

Everything that is written to the stdout is maintained with Heroku as an application log. Heroku is language independent, but the application logs are generated in the application-specific language. Each language offers some logging framework, which you can use. We will discuss some of them, but there are plenty of others. When choosing the logging framework, keep in mind that Heroku maintains logs from various sources. Your logging framework should be able to label log record with application identification (process ID, machine name) and timestamp. Also, the logging framework should maintain log verbosity (the severity levels such as into, debug, error are a matter of course).

There are many popular web server frameworks for python such as Flask, Gunicorn, or Django. Each of them offers its own logging utility. However, python offers native logging functionality and all these frameworks just extend it. As a result, all most popular python web servers offer similar logging functionality. We will demonstrate logging for the Flask (a lightweight Python web framework). You can use a simple logger for the routing in the Flask:

import os
from flask import Flask

app = Flask(__name__) # create Flask server
# get instance of logger and set log severity as defined by the cli parameter
gunicorn_logger = logging.getLogger('gunicorn.error')
app.logger.handlers = gunicorn_logger.handlers
app.logger.setLevel(gunicorn_logger.level)

@app.route('/')
def hello():
    app.logger.debug('This is a DEBUG log record.')
    app.logger.info('This is an INFO log record.')
    app.logger.warning('This is a WARNING log record.')
    app.logger.error('This is an ERROR log record.')
    app.logger.critical('This is a CRITICAL log record.')
    return 'Hello world!'

The app.logger generates log records with well-known priorities (debug, info, error, and others). The logger labels each log record with the timestamp and process ID. The executing of method hello() will generate the following log records (the method will be executed every time some user will request for the endpoint "/"):

[2021-06-02 10:08:37 +0200] [72137] [DEBUG] This is a DEBUG log record.
[2021-06-02 10:08:37 +0200] [72137] [INFO] This is a INFO log record.
[2021-06-02 10:08:37 +0200] [72137] [WARNING] This is a WARNING log record.
[2021-06-02 10:08:37 +0200] [72137] [ERROR] This is a ERROR log record.
[2021-06-02 10:08:37 +0200] [72137] [CRITICAL] This is a CRITICAL log record.

As you can see, the logger formats the log message into the usual plain text format. You can set up a log verbosity level with the CLI option when starting the server. You can run the previous python script with parameter --log-level=warning and the script will generate only log records with severity warning and higher.

If you want to use a dedicated logging framework then python offers, for example, Loguru.

Step 4 — Java Application Logging

Java offers a facade called SLF4J, which will allow us to integrate any Java logging framework at runtime. So the SLF4J is not a logging framework, but it implements any Java logging framework at deployment time. For demonstration, we will integrate the Java logging framework Log4J.

The Log4J defines the log template in the configuration file log4j.properties, which is located in the java project in the directory src/main/resources. For the purpose of the Heroku logging, the file log4j.properties can contain, for example, the following configuration:

log4j.rootLogger=ERROR,stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout.ConversionPattern=%d %5p [%t] (%F:%L) - %X{sessionId}%n - %m

The configuration defines that the logs are written to stdout, and each log record includes fields such as date(%d), severity level (%p), file and number of line in java code (%F:%L), sessions ID, message (%m) and others.

Now, we must bind Log4J and SLF4J together. You must define the following dependencies in the POM (see the further details about POM and Maven building in the official documentation):

<dependency>
    <groupId>log4j</groupId>
    <artifactId>log4j</artifactId>
    <version>1.2.17</version>
</dependency>
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-log4j12</artifactId>
    <version>1.7.30</version>
    <scope>test</scope>
</dependency>

The first dependency defines Log4J itself, and the second binds it with SLF4J facade.

At this point, you can import SLF4J dependencies into your java code:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Now, if you want to log something, you can create Log4J logger instance and call it with proper severity:

Logger logger = LoggerFactory.getLogger("MyLogger");
logger.info("This is an INFO log record.");
logger.error("This is an ERROR log record.")

The application dumps log records similar to the following snippet:

2020-08-17 03:57:57,100  INFO [main] (MyLogger.java:71) - 123456 - This is an INFO log record.
2020-08-17 03:58:02,767 ERROR [main] (MyLogger.java:72) - 123456 - This is an ERROR log record.

As you can see, the log record fields follow definition from the configuration log4j.properties.

Step 5 — C# Application Logging

There are multiple C# logging frameworks such as Log4Net, Nlog, or Serilog. All of them offer similar logging functionality. We will demonstrate the Log4Net framework.

The Log4Net set up the log record template in the XML configuration file. Consider the following configuration example:

<log4net>
    <appender name="A1" type="log4net.Appender.ConsoleAppender">
        <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%-4timestamp [%thread] %-5level %logger %ndc - %message%newline" />
        </layout>
    </appender>
    <root>
        <level value="DEBUG" />
        <appender-ref ref="A1" />
    </root>
</log4net>

The appender set up to ConsoleAppender means that the log writes to stdout. The output will be formatted using a PatternLayout where you can set up the pattern (timestamp - %timestamp, process name - %thread, verbosity level - %level, and others). The root logger is assigned to the minimal verbosity level DEBUG.

Following snippet of code demonstrates how to load XML configuration of Log4Net and create some log records:

using log4net;
using log4net.Config;

public class MyApp
{
    private static readonly ILog log = LogManager.GetLogger(typeof(MyApp));

    static void Main(string[] args)
    {
        XmlConfigurator.Configure(new System.IO.FileInfo(args[0]));

        log.Info("This is an INFO log record.");
        log.Error("This is an ERROR log record.");
    }
}

The XmlConfigurator parse the XML configuration file given as a command-line parameter. If you want to change logging behaviours then you change only XML configuration and you do not need to recompile code.

If you execute this code with XML configuration as a command line parameter then the following log output will appear in your console:

2021-05-05 14:07:41,508 [main] INFO  MyApp - This is an INFO log record.
2000-05-05 14:07:41,529 [main] ERROR  MyApp - This is an ERROR log record.

As you can see, the log record fields follow definition from the XML configuration.

Step 6 — Node.js Application Logging

The typical use case for Node.js in server-side is a web application layer, where the REST API routing happens. For this purpose the framework Express.js is the most popular. In the area of REST API, you typically want to log the API requests. The framework Morgan aims specifically for logging only the HTTP requests. The usage is very simple. Let's view the following simple Express.js application with Morgan logging included:

const express = require('express');
const morgan = require('morgan');

const app = express();

app.use(morgan('combined'));

app.get('/', (req, res)  => {
  res.send('Hello from Express.js server!')
});

// run the server on the port 8080
app.listen(8080, () => {
  console.log('Hello world!');
});

The application has only a single URI route "/" and it will generate the log record every time this route will be requested. Morgan logging is enabled by the statement app.use(morgan('<verbose>')), where the <verbose> determines format and verbosity of the log record. For the purpose of the Heroku logging, the two possible verbosity levels are considerable: common and combined.

Standard Apache common log output template:

:remote-addr - :remote-user [:date[clf]] ":method :url HTTP/:http-version" :status :res[content-length]

Example:

::1 - - [02/Jun/2021:11:43:19 +0000] "GET / HTTP/1.1" 304 -

Standard Apache combined log output template:

:remote-addr - :remote-user [:date[clf]] ":method :url HTTP/:http-version" :status :res[content-length] ":referrer" ":user-agent"

Example:

::1 - - [02/Jun/2021:11:40:52 +0000] "GET / HTTP/1.1" 200 29 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"

As you can see, Morgan uses Apache common format, which is a well-known standard and many utilities for log manipulating can handle it.

If you want to log some other application behaviour than routing, you can use some other general-purpose Node.js logging frameworks: Winston, Pino, or Loglevel.

Step 7 — PHP Application Logging

There are various logging frameworks for PHP. The built-in PHP error log is automatically sent to stderr, and any message logged using the error_log() function will be available in Heroku logs:

error_log("This is a error log.");

However, if you use some web application server framework then it usually includes some built-in logging. Nowadays, the framework Laravel is very popular as a simple monolithic web application framework. Laravel project tree includes configuration of the various logging channels in the file app/config/logging.php. By default, the configuration file contains the variable LOG_CHANNEL that set up a log channel. You can simply set the LOG_CHANNEL variable to value errorlog for correct logging on Heroku:

$ heroku config:set LOG_CHANNEL=errorlog

Heroku CLI command config:set set the configuration variable LOG_CHANNEL to the given value and your logs will be maintained by the Heroku.

Laravel offers a logging facade in the namespace  Illuminate\\Support\\Facades, so you must include it in the source code. Now, you can use Log facade methods with common priorities (debug, error, info, critical, and others):

<?php

use Illuminate\\Support\\Facades\\Log;

Log::info('This is a INFO log.');
Log::error('This is a ERROR log.');

The snippet of code demonstrates how to use log static functions. You can view all available methods in the documentation. If you execute this code, you will see log records similar to the following example:

[2021-05-01 14:16:39] production.INFO: This is a INFO log.
[2021-05-01 14:16:40] production.ERROR: This is a ERROR log.

As you can see, the log record includes timestamp, log level severity and log message.

Conclusion

In this tutorial, you maintained application logs from the different sources with the Heroku platform. You viewed and filtered the logs with the Heroku CLI.

Next, you created the application logging for your application in the platform-specific language (Python, Java, C#, JavaScript, and PHP) and configured it for the Heroku logging.

Article by

Jenda Tovarys

Jenda leads Growth at Better Stack. For the past 5 years, Jenda has been writing about exciting learnings from working with hundreds of developers across the world. When he's not spreading the word about the amazing software built at Better Stack, he enjoys traveling, hiking, reading, and playing tennis.

Got an article suggestion? Let us know

Next article

How to Start Logging with Postfix

Learn how to start logging with Postfix and go from basics to best practices in no time.

→

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.