After a successful installation of Docker, the default configuration is used to run the docker daemon. In a development environment this may not be a major issue, but in a production environment it is desirable to use custom configuration. System administrators configure the daemon to run according to specific requirements of an organization. Process managers like SysVinit, Upstart, or systemd are configured to run the daemon. Directly running the docker daemon is possible using the command dockerd. In this setup the docker daemon listens on socket unix:///var/run/docker.sock. To check the socket your daemon is listening on, run the dockerd command at a terminal.
When you decide to run the docker daemon via dockerd instead of a process manager, you pass configuration options through the docker command. Here we will demonstrate the use of three flags to show how configuration options can be passed. These examples are just used for demonstration purposes. There are many options that can be used with dockerd, for an exhaustive list please refer to documentation available here https://docs.docker.com/engine/reference/commandline/dockerd/. The -D or –debug accepts a true or false option to set debugging to true or false respectively. By default this option is set to false. The -H or –hosts flag is used to pass a socket or list of sockets for connecting. The –tls flag can be set to true or false to enable or disable TLS. By default TLS is set to false.
When debug is enabled using the -D flag the system administrators are able to gather a lot of information on how the daemon is running. The Docker daemon log contains commands that are received via the remote API and events that are as a result of normal operations of the daemon. When interacting with the daemon via the remote API; commands, warnings, or errors resulting from the commands are automatically logged.
The information that is captured for each event is listed below.
- The version of the API
- The time stamp the event occurred
- The type of request i.e. if it is a GET, PUT, POST or any other
- Detailed information on the request and the return
- The level of logging i.e. if it is information, warning, or error
- The end point of the event i.e. if it is a container, an image or a data volume
Logging information on daemon events shows the information listed below.
- The time stamp when the event occurred
- The level of logging
- Detailed information on the event
Daemon events are useful for providing information on the state that the containers are in.
Log events are generated and formatted using logrus which is a Go programming language logging approach. On ubuntu the log is located at /var/log/upstart/docker.log. On CoreOS, SUSE, Fedora, CentOS and Red Hat Enterprise the log is located at journalctl -u docker.service. When docker is started at a terminal via docker daemon, command logging is displayed on the terminal. The recommended way of running docker is as a service. The logging formats are different when docker is started from a terminal and when started through a process manager. When the daemon is started through a process manager the logfmt format is used. The logfmt format uses a key-value pair. There is less structure in command line logging. When the docker daemon runs as a service the lofmt logging format is used. A logging entry begins with a UTC time stamp, then the log level and finally the message. For example a log entry of starting a container will appear as shown below.
time=”2016-11-03T16:35:15.290647728-05:00″ level=info msg=”POST /v1.21/containers/create?
On operating systems that use systemd additional information on time stamp, hostname and the process that resulted in the log entry. For example, on Debian the previous log entry on starting a container would appear as shown below
Nov 03 16:35:15 virtualbox-debian docker[2990]: time=”2016-11-03T16:35:15.290647728-05:00″ level=info msg=”POST /v1.21/containers/create”
Daemon log information generates a wealth of insights on service operations that cannot be captured with container level logging. Detailed information on how containers interact, Docker service and host machine. By gathering this information you are able to understand container lifecycle and any disruptions that arise. For example, when a container shuts before giving a warning there is no logging that happens so the docker log command or the application logging cannot be relied to identify the cause of the problem. The work around this problem is to use time stamp or container ID to search through the daemon log. An effective logging strategy includes implementing logging at the application, at the operating system logging and at the Docker service level.
Application level logging is appropriate when a developer would like to leverage existing logging framework and avoid the need to implement host logging. Application logging has the limitation of losing logs when a container shuts down. To overcome loss of logs with a container shut down you can send the logs to a remote location or configure a data volume that is persistent. Another challenge with application logging is when you have many identical containers that need a unique way of identification.
When using application level logging, data volumes are a safe way to prevent loss of information with a container shut down. You implement this by mapping a container directory to a directory on the host operating system. A single volume can be used to implement logging across many containers but moving containers to different hosts becomes problematic because you cannot track all the data. Data volumes are effective when implementing a long term centralized logging strategy. Because logs are available on host operating system you can backup and access the logs from different containers.
Another approach to logging is to use a container specifically set up for logging. This avoids the shortcoming of relying on a service running on the host, you are able to manage logging from within Docker. The logging container captures log information from other containers and processes them as required.
In this tutorial, we introduced logging as a way of gathering information on the status of Docker daemon and other Docker services. We discussed how logging happens when docker daemon is started from the terminal and when a service manager is used. We also discussed logging approaches that can be used.
