How to monitor your Docker infrastructure with Checkmk

Checkmk is a monitoring solution that allows you to monitor almost any aspect of your IT infrastructure. Recently, tribe29, the company behind Checkmk, announced the official monitoring plug-in number 2,000, and thus, Checkmk provides an out-of-the-box solution for most network devices, servers, applications, cloud environments, but also for more modern IT infrastructure assets such as Docker and Kubernetes.

In this tutorial, I want to show you how to run Checkmk in a Docker container, and will also monitor a Docker container with Checkmk. To do so, I will use the Checkmk Free Edition, which allows you to monitor up to 25 hosts for free. As an alternative, you can also use the Checkmk Raw Edition, which is 100% open source and supports an unlimited number of hosts.

An important principle of Checkmk is that the monitoring always adjusts to the needs of your infrastructure – not the other way around. Deploying Checkmk in Docker is not a must, you can run Checkmk as a Linux server or as an appliance, as well. However, running applications in Docker has several benefits, especially in environments that mostly rely on containerized applications, and thus, you also should be able to host your monitoring accordingly.

Step 1: Preparation

For this guide, I will use a hardware server running on Ubuntu Focal 20.04 as my Docker host. I will monitor my Docker containers with the Checkmk Linux agent and the Checkmk Docker plug-in. To run that plug-in you have to have a Docker Python library version 2.0.0.0 or higher installed. You can check with the command ‘python3’ in the terminal.

If you have an outdated version, use this command to install the Docker Python library:

sudo apt update
sudo apt install python3-pip uidmap

It is important that the packages docker-py or python-docker-py are not installed. These make an outdated and incompatible version of the Docker library available under the same namespace. Use the command below to uninstall them and then re-install the Docker Python library.

sudo pip3 uninstall docker-py docker

You also need to have the Docker engine installed to run Docker containers. The Docker documentation shows you how to do that for several operating systems, including Windows and macOS. Thus, if you want to run Checkmk on these operating systems, you can do so with Docker.

## Scripted installation  for Dev ###
curl -fsSL get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo su -

You also need a client to download Checkmk and send the package to your host server. The client must have root access to your monitoring host server. If you have that, you are good to go. Checkmk does not need a separate database and is generally fairly resource-efficient. Especially, if you use the Checkmk agents, you can easily monitor a few thousand hosts with a single instance. In my case, my host server has two CPU cores and eight gigabytes of RAM, more than enough for a small monitoring environment.

Step 2: Download and install Checkmk

This tutorial is based on Checkmk version 2.1.0. To get the latest version, you can check the Checkmk download page.

• Open the terminal on your Docker host, become root and install Checkmk with the command:

docker container run -dit -p 8080:5000 \
  --tmpfs /opt/omd/sites/cmk/tmp:uid=1000,gid=1000 \
  -v monitoring:/omd/sites --name monitoring \
  -v /etc/localtime:/etc/localtime:ro \
  --restart always \
checkmk/check-mk-free:2.1.0

See below screenshot for how the installation should progress.

Besides creating a container named ‘monitoring’, you also have tied the data from the monitoring site in this container to a persistent location in the Docker node file system. You can adjust the parameters, of course, if you wish. Your Checkmk container should be up and running, but do not leave the terminal yet. For the first access via the web interface, you will need to know the randomly-created password.

• You find that in the logs, open them with this command:

docker container logs monitoring

See below screenshot.

• Copy your password.
• Open the link ‘your_server/monitoring’ on your browser to access Checkmk.
• Add ‘cmkadmin’ as username, and use the password you copied before.

Step 3: First steps in Checkmk

There you are, you just installed Checkmk. You are now in an empty Checkmk environment. Before you go ahead, do not forget to change your password.

• Go to User -> User profile -> Change password.
• Add your old password and change it to a new one.

Step 4: Set up rules for Docker and DCD in Checkmk

Checkmk is using the agent plug-in mk_docker.py to monitor Docker containers. This is really powerful in combination with the Agent Bakery and Dynamic Host Configuration of Checkmk. The Agent Bakery allows you to adjust Checkmk agents to your needs and update them automatically. The Dynamic Host Configuration is based on the Dynamic Configuration Daemon (DCD) and uses information from Docker hosts (and also AWS, Azure, Kubernetes, VMware, and other sources) to create and remove hosts automatically. Thus, with so much automation power, you just need to set things up once and will have an easy and scalable monitoring for all your Docker containers.

The Agent Bakery and the DCD are features of the Checkmk Enterprise Edition. If you use the Checkmk Raw Edition, you have to configure the Checkmk agent and the Docker hosts manually.

• Go to Setup and search for ‘docker’. Click on the agent rule ‘Docker node and containers’.
• Click Add rule.
• By default, Checkmk will use the first 12 characters of the container ID to name the automatically-created hosts in Checkmk. You do not have to change anything, but I decided to adjust that under Host name used for containers and tell Checkmk to use the container name instead. It is just nicer to see the names of containers.
• Under Conditions, you can decide how to apply the rule. I will tie this rule to the main directory. Thus, I do need to make any changes.
• Click on Save.

Next, you need to take care of the DCD:

• Go to Setup, search for ‘dynamic’ and click on ‘Dynamic host management’.
• Click on Add connection.
• Under General Properties, choose a Unique ID and a Title. I went for ‘docker_cmk’ and ‘piggyback’.

• For the Connection Properties, you have to click on ‘Add new element’, otherwise Checkmk cannot automatically create hosts. Under Piggyback creation options, I left ‘Main’ as the target folder for the automatically created hosts.
• Besides that, you can leave everything at the default value, but you can adjust several things.
• Click on Save.

You literally just added two words, but there is a bit more to unpack on this page. With the Docker agent plug-in, the Checkmk agent will not only gather information about your Docker host, but also about the Docker containers running on it. These details will be saved as piggyback data by the Checkmk agent. The piggyback mechanism of Checkmk allows querying data from a host over another host. The Dynamic Host Configuration of Checkmk will fetch that data and create a host for each container with piggyback data available.

You have several options to adjust that process. The Sync interval decides how often Checkmk should check for new data. The default value of one minute is fine for my Docker environment, the piggyback data is also just written once per minute. But Checkmk supports lower check intervals, which is great in very dynamic environments. However, this also results in a higher CPU utilization on the Checkmk server.

You can also tell Checkmk to take out hosts of containers, if there is no data available for a while (by default ten minutes) with the checkbox Delete vanished hosts. I did not change it here, but keep this option in mind for the future.

By default, Checkmk only shows you the most important configuration options. If you click on ‘show more’ on the right, you see more advanced options, but you do not have to change any of them for this tutorial. I mention this, because if you click on ‘show more’ in this case, you see the option ‘Hosts attributes to set’ with the attributes that Checkmk will set for all hosts created with this rule. In case you change these attributes, they will be changed for all hosts during next execution of the connection. Be aware, that you cannot adjust attributes for an individual host created with the DCD in the Checkmk host menu, but always have to adjust the attributes here in the rule itself.

Step 5: Bake and install the Checkmk agent

You might have recognized the highlighted field with the yellow exclamation mark (!) in the top right corner. Your changes in Checkmk will not affect your operational monitoring directly. You have to confirm the changes.

• Click on the highlighted field to see the ‘pending changes’.
• Now, click on Activate on selected sites to confirm. Your monitoring rules are now applied.

Next, you have to use the Agent Bakery to bake an agent and install it on your Docker host:

• Go to Setup -> Agents -> Windows, Linux, Solaris, AIX.
• Click Bake agents and wait for the Agent Bakery to create a new agent.

• Your new agent should appear. Download the package that matches your Docker host server. For me, that would be the DEB package.
• Install the baked agent on your Docker host server.

Step 6: Add your Docker host and monitor your Docker containers

Now, you just need to add your Docker host and Checkmk will do the rest:

• Go to Setup -> Hosts and click on Add host.
• Add a Hostname and the IPv4 address of your Docker host server. In my case, The name is ‘example.docker.host.net’.
• Click on Save & go to service configuration.

Checkmk will now scan for services. As you see, Checkmk detected several Docker services and also a Docker container. This is the container Checkmk is running in. You will not only monitor your containers but also your images, volumes, and more details.

• Wait until that is done and click on Accept all. 

The service for the Checkmk agent is on WARN, because I did not set up TLS yet. tribe29 recommends encrypting the data and gives you this warning. You find an overview of how to improve the security of Checkmk in this chapter of the Checkmk guide.

• Confirm your ‘pending change’ again by clicking on the field in the top right corner. You now see that you are actually about to add more than just one host to your monitoring. This is the DCD adding hosts based on piggyback data.
• Click on Activate on selected sites to accept the changes.
• If you go to all Monitoring- > All hosts can see a list of all the hosts in your monitoring. I see my Docker host, but also my container ‘monitoring’ that Checkmk has created automatically.
• If I click on a host, Checkmk shows me the services of this host. My container ‘monitoring’ looks good:

Step 7: Using the Checkmk agent within a container (optional)

You can also run the Checkmk Linux agent within the container itself and then add the container as an independent host to your monitoring. This allows you to monitor details in the container such as running processes, databases, or log files but is some extra work because the host is not automatically created for you.

In the Checkmk container, the Checkmk Linux agent is already installed by default. Thus, you can add the ‘localhost’ as host and pull the data directly from this container.

• Go to Setup -> Hosts and click on Add host.
• Add ‘localhost’’ Hostname.
• Click on Save & go to service configuration.
• Wait for service discovery and click Accept all. 
• Click on Activate on selected sites to accept the changes.

The ‘localhost’ is now in your monitoring as a separate host. You monitor the same information as in the host ‘monitoring’, the difference is the way, the information is transported. In the case of the host ‘monitoring’, Checkmk gets the piggyback data from the Linux agent running on the Docker host. For the host ‘localhost’, the data comes directly from the Linux agent running in the Docker container. You see the difference in the first service ‘Check_MK’ which indicates how the data has been pulled.

For the Checkmk container, this does not add too much value, but the additional information is great if you run your own applications in containers. If you want to use the Checkmk agent in your container, you need to implement the agent and the directories into your Docker container. Please note that the agent only works if all necessary commands are also present in the container. Especially with minimally-built containers based on Alpine Linux, it could very well be that elemental things such as ‘bash’ are not present.

Summary

This tutorial ends here. Checkmk will continue adding all containers that will be created on your Linux host. Thus, you can, but do not have to spend more time on exploring Checkmk, and can also just go back to spending more time on doing other things. If you want to dig deeper into monitoring, I recommend this page about server monitoring. It gives you a neutral guide on the topic and is a great starting point.