Logging for Docker

When moving from a development to production environment, it becomes important to log commands issued to Docker for auditing purposes. Some reasons for doing so are:

  • compliance to government requirements,
  • auditing and tracing of issues,
  • requirement of industry standards,
  • client/end-user requirement,
  • and so on.

Docker offers two types of logging mechanisms – Containers, and Docker daemon. The logs are always for one layer down, which means your Container will store logs for your application events, and the Docker daemon will store logs for your Container events.

Logging by Containers for Application

By default, Containers will use the json-file logging driver to log your application. This driver writes JSON messages to file. You can use the following command to view the logs:

$ docker logs <CONTAINER_NAME>

If you choose not to use the default json-file logging driver, you can choose from the list of supported drivers from the Docker documentation (https://docs.docker.com/engine/admin/logging/overview/)

Logging by Docker Daemon for Containers

The Docker daemon also streams events regarding its Container. You can stream the events to your SSH session via the following command:

$ docker events

A simple docker run generates the following logs:

2016-06-30T11:13:17.498875365+08:00 container create 5f8f5083d0f4943e105d89ad19fc6147bd0b2b6360659a14e9d4042fb39d330b (image=r-base, name=silly_hoover)
2016-06-30T11:13:17.500661709+08:00 container attach 5f8f5083d0f4943e105d89ad19fc6147bd0b2b6360659a14e9d4042fb39d330b (image=r-base, name=silly_hoover)
2016-06-30T11:13:17.538882264+08:00 network connect eb995a62baddfb668c9a14196738e6c92f77e99b107aa85a92af5568d7a9b265 (container=5f8f5083d0f4943e105d89ad19fc6147bd0b2b6360659a14e9d4042fb39d330b, name=bridge, type=bridge)
2016-06-30T11:13:17.622084696+08:00 container start 5f8f5083d0f4943e105d89ad19fc6147bd0b2b6360659a14e9d4042fb39d330b (image=r-base, name=silly_hoover)
2016-06-30T11:13:17.623051766+08:00 container resize 5f8f5083d0f4943e105d89ad19fc6147bd0b2b6360659a14e9d4042fb39d330b (height=44, image=r-base, name=silly_hoover, width=168)

This command streams the Container events, but takes over your terminal. To stream all the events into a file, use the following command:

$ docker events > <LOG_FILE> 2>&1 &

The “docker events > <LOG_FILE>” streams the standard output (stdout) from the command to the file specified by <LOG_FILE>. The 2>&1 indicates to also stream the standard error (stderr) of the command to the <LOG_FILE>. The last “&” is to indicate that this should run in the background.

Running the command should give you a pid of the job:

$ docker events > log.dock 2>&1 &
[1] 47701

Running a “ps” validates that:

$ ps
PID TTY TIME CMD
46828 pts/0 00:00:00 bash
47701 pts/0 00:00:00 docker
48480 pts/0 00:00:00 ps

To stop the streaming, issue a kill command:

$  kill <PROCESS_ID>

 

Logging by Unix for Docker Daemon

Having access to the Docker daemon is a powerful thing – Docker mentions that “only trusted users should be allowed to control your Docker daemon” (https://docs.docker.com/engine/security/security/)

While you may trust your users, or enforce some processes, there may be times where it might be prudent to log down any commands issued to the Docker daemon for auditing and tracking.

Another scenario is when you are running a Docker Swarm cluster on top of several compute machines used by several teams. Having access to the Swarm manager means that the “trusted user” from Team A can essentially start/stop/remove Team B’s Images, Containers, and even data volumes – be it accidental or purposefully. When the unfortunate happens, it helps if there is a log file.

To log all commands sent to the Docker daemon, auditing by the Unix system is one possible way. The “auditd” Audit daemon is a useful tool that can achieve this, and can be installed by doing a package install.

There are two files (rules and config) that the auditd daemon uses, and depending on your distro, they may be stored at different locations. The most likely location is “/etc/audit”, and the main gist is as follows:

  • auditd.conf – this stores the audit daemon configuration
  • audit.rules – this stores the audit rules for auditd
  • rules.d/audit.rules – this is the drop in audit rules
  • /var/log/audit – this is where all the logs are stored

To have auditd to log all commands issued to the Docker daemon, add the following Filesystem rule:

-w /usr/bin/docker -p rwxa -k docker

This Filesystem (-w) rule tells auditd that we wish to watch the Docker daemon (/usr/bin/docker) for the following permissions: read, write, execute, and attribute change. The -k flag is an optional key that is added to the logs to help identify which log entry matches which rule.

After adding the rule, restart the auditd daemon to load the new rules:

$ sudo service auditd restart

 

Conclusion

  • Logging is important for a variety of reasons
  • Logging is always one layer down – Docker has built in logging features for your application, and containers
  • As the time of post, you need to use an external logging mechanism (e.g. auditd) to capture commands passed to Docker daemon

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s