After applications have been developed and deployed on production systems problems are inevitable, so as a Docker developer you need to be prepared to handle such situations. Debugging applications that have been containerized is very similar to debugging ordinary processes. In this tutorial, we look at different options that are available to a developer to help understand what is going on in a container.
Docker has a docker top container that was specifically developed to get information on processes that are running inside the container. This command is very useful because besides running on local hosts it can run on remote hosts as well. The docker top command is one of the many ways to know what is going on inside the container. To use the command, you have to specify the ID of the container as returned by docker ps. In the Docker images tutorial, we demonstrated how to search and download images on Docker hub. In that tutorial, we downloaded a bitnami image, please refer to that tutorial to learn how images are downloaded. We check if the image is available on our host by running docker images. Once we confirm we have the image locally, we create a container from it using the command below.
sudo docker run –name=firstdebugcont1 -p 80:80 -p 443:443 bitnami/wordpress
Once we have started a container, we use docker ps to return its image. From the output below, we have an image with name firstdebug which is what we specified when creating the container.
Get the container ID and pass it to docker top command as shown in the command below:
sudo docker top 4f0be8e5bf64
The docker top command returns an ordered list of what is running inside our container.
Using docker top is just one of the ways to know what is running but it is not the best. One shortcoming when using docker top command is the namespacing of user ID and file systems. A case in point is when we have a user specified within the docker container, but we do not have the user specified on the local host. It may then happen that we have two containers referencing the same ID or an ID that is not even the ID of the host system.
An alternative way of identifying processes that are running is to use Linux ps. This will return all processes running on the host whether or not they are containerized. Linux ps is useful for debugging because it shows you all processes that are running in Docker. When we run ps without any options, the output is not very useful because it just shows processes being run by all users and the terminal session. When we pass the aux option using ps aux, we get all processes from all users irrespective of their terminal. To get a tree view of processes we pass the axlf option like this ps axlfww.
The ps axlfww comand will show you the Docker daemons and associated containers that are running on your host.
To return only the Docker tree use pstree command and pass the user associated with Docker. This command does not return PID so it is useful when you have very many containers running on your host. Another useful debugging command is strace which helps you understand how a process is interacting with the Linux kernel. The command records all process activities so you are able to get a lot information. You pass the process id to strace. To get the process id use pidof command.
Before you can use strace you need it installed. Use the commands below to install and start strace.
sudo apt-get install strace
sudo strace -p 2968
To check resources, a process needs items such as sockets and files are working properly you use the lsof command by passing the process id. An example is shown below
sudo lsof -p 2968
Inspecting processes is easy but is not the case when you need to debug containerized applications that are running over a network. Containers that are not running with a host networking enabled will not be displayed on netstat output because they will be allocated their own ip addresses. If we run netstat, we will get results that we would be expecting
sudo netstat an
The netstat command will show the ip address and the port the container is using. When using host networking, it is possible to identify the process that has been bound to a specific port. However when using a proxy, netstat will not show you the container that is bound to a specific port, the docker-proxy will always be shown. To identify the port on which a container has been bound to, you can use the docker ps command.
Another useful command for debugging containers running over a network is tcpdump. The command helps you understand network traffic. Options available with this command are broad, helping you identify packets on specific ips, ports and those of a certain size.
In this tutorial, we introduced a situation where an application deployed on a production system encounters problems. When this situation happens we need to understand what is happening with our container. Docker and Linux have tools to help a developer debug problematic applications. In this tutorial, we discussed the available docker tools. We also discussed available tools in Linux to debug containers running as local processes and containers running over a network.
