We defend the port. How to protect infrastructure on Docker with minimal effort

Docker is a great thing that can save a lot of time and effort. In this article, we will talk about how to use Docker as securely as possible and catch potential threats in advance. You will also find here ready-made recommendations, commands and tools that are easy to use in your project.

Docker-based container virtualization is extremely popular among DevOps admins. It allows you to quickly roll out software development and testing tools or entire projects from many components. And everything would be fine if not for a whole class of specific security threats. This is the ability to escape from the container to the host system, and the instance shutdown as a result of a “denial of service”, and cases with the exploitation of OS kernel vulnerabilities, and tossing pre-compromised reference images into the repository, and similar attacks.

In this article, with the help of admin ingenuity, direct hands and all available standard Linux tools, we will learn how to set up the most secure and controlled DevOps environment based on Docker containerization.

Through the eyes of a security guard

We have already talked about the containerization system itself more than once. However, the issue of configuring security options for containers has remained on the sidelines all this time. Now we will look at Docker through the eyes of a security specialist (or a hacker, if we are serious about this topic).

What is Docker and how it works

Docker is software for automating application deployment and management in containerized environments. The tool allows you to pack any application with all its environment and dependencies into a special container that can be run on any modern Linux.

Thus, each container we create already includes everything necessary for the application to work — libraries, system tools, code, and runtime. With Docker, DevOps administrators and developers can quickly deploy, scale, and run their applications in any environment, while being confident that the code they write will run smoothly.

Docker has many of the features that traditional virtualization systems have. For example:

• independence — the container can be moved to any OS with a previously raised Docker service and run with one command;
• self-sufficiency — the container will perform all the functions assigned to it anywhere, wherever it is launched, without additional configuration and maintenance.

However, unlike traditional virtualization, where we usually create the image ourselves, Docker works with images taken from repositories. There are public and private repositories of official and unofficial images. In the documentation, they are called docker registry. And the most popular and frequently used repository today is Docker Hub .

Docker image is a sequential set of software layers. Each layer is the output of a command in a Dockerfile configuration file . That is, the image is a template based on which we launch the container. But everything that is launched based on this image is the container itself (instance). As a result, Docker has a very useful feature that allows you to run several identical copies of this image, that is, containers, from one prepared image.

Common security issues associated with Docker

Let’s go through typical cases related to Docker infrastructure security.

Host System Security

One of the simplest and at the same time key Docker security issues is the security of the host machine, in particular the OS kernel. Indeed, in an already compromised system , isolation and other container security mechanisms that we could use are unlikely to help us. This is because Docker is designed in such a way that all running containers use the host kernel. Therefore, the host should be a patched, updated Linux distribution without known vulnerabilities and signs of malware infection.

A classic example of such a case is escaping from a Docker container. In the official documentation, this bug is called “container breakout” and describes a situation in which a program running inside a container manages to overcome isolation mechanisms and gain root privileges or access to important information stored on the host. A typical example of this bug is DirtyCow (CVE-2016–5195), we have already talked about it .

To implement protection against such situations, a rule is used to reduce the number of privileges for a container that are granted to it by default. So, if the Docker daemon is running as root, then we create a user-level namespace for it with minimal privileges.

Resource exhaustion, or DDoS on a container

If we compare containers with virtual machines, then the former are more lightweight. Even on old and weak hardware, you can run many containers. However, daemon configuration errors, network stack errors, architectural design flaws, and hacker attacks can lead to Denial of Service conditions.

For example, a certain container or a whole pool of containers can devour all the CPU time of the host and squander its performance. A similar situation can occur with network interfaces when the number of generated packets exceeds the normal network bandwidth. But the way out here, in principle, is quite simple: properly set the resource limits for the container.

Compromised images for Docker

Docker is distributed as open source, and images for creating your own containers are also freely available to the public. This means that attackers using phishing and other social engineering methods can try to manipulate the actions of users of Docker Image repositories in order to force them to download a pre-compromised image with malware or backdoors.

For example, in April of this year, a lot of noise was made by the news that unknown attackers gained access to the database of the world’s largest image library for Docker Hub containers. As a result, usernames, password hashes, and tokens for the GitHub and Bitbucket repositories used for automated Docker builds were compromised. Or take the news from the summer of 2018, when seventeen container images containing dangerous backdoors were removed from the official Docker Hub registry. As experts later found out, malware in the form of cryptominers penetrated users’ servers through them and reverse shells were used.

Strengthening the security of Docker

So how do we harden Docker? First, we’ll look at general recommendations, and then specific examples of configuring rules and security policies for a container pool.

Use of authentic images

Let’s start with a common problem for Docker repositories — the reliability of the image. To avoid problems with fake images, use private (private) or strictly trusted repositories (trusted repositories) like Docker Hub . Unlike other repositories, the images stored there are always scanned and viewed by a special security bot called Docker’s Security Scanning Service .

We verify images through the Docker Content Trust service

Another useful and accessible tool for everyone to use is Docker Content Trust . This is a new feature available since Docker Engine 1.8. It allows you to verify the owner of the image. Thus, this service protects you from fakes and forgeries, replay attacks and key compromises.

Host and container verification with Docker Bench Security

A very useful tool that I myself have recently used is Docker Bench Security (see also its documentation ). In essence, this is a large collection of recommendations, tips and practices for deploying containers in production. The tool is based on the recommendations in The CIS Docker 1.13 Benchmark ( PDF ) document and is applied in the following areas:

• host configuration (kernel, processes, permissions);
• Docker daemon configuration (network, RAM allocation, CPU);
• Docker daemon configuration files
• container images and build files;
• container runtime;
• Docker security default options.

To install this verification script, clone the repository with the following command in the terminal:

$ git clone git@github.com:docker/docker-bench-security.git

After that we run the script:

$ cd docker-bench-secutity

and use this long command:

$ docker run -it --net host --pid host --cap-add audit_control \
  -e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
  -v /var/lib:/var/lib \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /etc:/etc --label docker_bench_security \
  docker/docker-bench-security

After completing these steps, you will build all the containers and run a script that will check the security of the host machine (kernel OS) and its containers. And it only takes a few minutes!

Using native security options on host OS

To increase security, you can use regular Linux tools. Among them are AppArmor, SELinux, grsecurity and Seccomp.

The easiest way would be to download the default Seccomp profile for Docker. To ensure the most banal level of security, it is enough to adjust the white list of system calls around line 52 — that is, simply remove the risky commands chmod, fchmodand from it fchmodat.

However, fine-tuning these security mechanisms is far beyond the scope of this article. If you have never configured them yet, refer to the official documentation. From myself I recommend a couple of good articles on Seccomp and SELinux .

Configuring Hardening Options for Docker

So we finally got to the console! Now we will build security for Docker with handles. Ready? Go!

Limitations of Docker Engine

Recall that the Docker Engine is a kind of API that listens for all incoming requests and interacts with the host’s underlying engine. Docker Engine communicates on three sockets: unix, tcpand fd.

The default safe state is listening on a unix socket. To activate this option, type in the terminal

$ dockerd -H "unix:///var/run/docker.sock"

Done, nothing else needed!

Set a limit on execute privileges

If possible, run containers as a normal non-root user:

$ docker run -d -u 1000 debian sleep infinity
$ ps aux | grep sleep
1000 ... sleep infinity

To minimize the area of ​​potential attack, consider the following questions:

• What network connection is required for the application to work, and is it required at all?
• Does the application need direct socket access?
• Does the application need to send and receive UDP requests?

To set the restrictions, we will use the --cap-drop andand commands --cap-add.

Suppose an application absolutely does not need to change system processes or bind privileged ports, but it does need to load and unload kernel modules. Relevant capabilities can be removed and added as follows:

$ docker run \
  --cap-drop SETPCAP \
  --cap-drop NET_BIND_SERVICE \
  --cap-add SYS_MODULE \
  -ti /bin/sh

You also need to watch out for mounting potentially dangerous host resources (for example, such as /proc, /dev, /var/run/docker.sock). Despite the fact that these resources are necessary for the operation of containers, it is still worth considering the need to restrict the access of processes to them. For example, it will be enough just to set the read-only access mode to them.

To secure network communication, you can configure iptables rules implemented in Docker. For example, specify a range of IP addresses for the source of packets to restrict traffic to a container. This will help prevent a deeply isolated container from accessing the external network. You can enable this feature with just a couple of commands:

$ iptables -t filter -A FORWARD -s
<source_ip_range> -j REJECT --reject-with
icmp-admin-prohibited

Limiting the consumption of system resources

To calibrate the pool of resources allocated to a container (CPU, RAM, SWAP, I/O), you need to set thresholds with the following commands:

• -m, --memory— set the RAM allocation limit (hard);
• --memory-reservation- set a soft memory limit (soft);
• --kernel-memory- set the kernel memory limit (kernel OS);
• --cpus- limit the number of CPUs used;
• --device-read-bps- Limit read bandwidth for a specific device.

Also don’t forget to use a special feature called cgroups. Control groups (cgroups) are a standard tool provided by the Linux kernel that allows you to restrict the access of processes and containers to system resources. Some limits can be controlled directly from the Docker command line, for example:

$ docker run -it --memory=4G --memory-swap=1G debian bash

With this command, we allocate 4 GB of RAM for the container and 1 GB for caching in swap.

Monitor container activity

Monitoring, detection of anomalous, suspicious container behavior and notification about it is one of the key tools for analysis and timely response to incidents. For these purposes, you can use the open source tool Sysdig Falco .

Sysdig Falco works as an intrusion detection system and is especially useful when using Docker because it maintains container-specific context such as container.id, container.image, Kubernetes resources or namespaces during rule creation.

Oh, by the way, another thing from the standard set of OS control and audit is event logging. To do this, we launch the container with the key responsible for logging:

$ docker run -v /dev/log:/dev/log
<container_name> /bin/sh

Burning CVE vulnerabilities in containers

Docker containers are essentially isolated black boxes that run on a parent host. However, everything can work fine, but at the same time, vulnerable software is inside. Moreover, upstream vulnerabilities can be patched long ago, but not in your local image! And if you do not take action, such problems can go unnoticed for a long time. And what, pray tell, do with it?

Many Docker image registries offer scanning of these very images. For example, the CoreOS Quay service uses an open source Docker image security scanner called Clair . Clair is an application written in Go that implements a set of HTTP APIs for uploading, downloading, and parsing images. Vulnerability data is downloaded from various sources such as Debian Security Tracker or Red Hat Security Data. In this case, the Clair engine works like a static analyzer without launching a container for execution.

Well, if you are a completely green admin or the management is hellishly squeezing money, somewhere on the Web, they say, there are similar services with the same capabilities, but at the same time completely free! True, I have not yet met such. Therefore, if there is absolutely no choice, you can use the good old OpenVAS or a similar security scanner, which you will run manually according to a schedule within your corporate grid.

WWW

Some interesting practical guides, tutorials and cookbooks on how to harden Docker security for self-study:

• Mitigating Docker Security Issues — A guide that looks at the main risks and threats to Docker security. There you will also find practical examples of security settings to solve these problems;
• Docker Secure Deployment Guidelines — a small page on GitHub describing recommendations and basic security settings for Docker;
• 10+ top open-source tools for Docker security — An overview of several useful tools for controlling the security of source images and running containers.

Conclusion

Congratulations! Today we improved your admin skills in protecting containerization systems based on Docker. Now you know about the main threats to the DevOps environment and understand how you can pay for carelessness if you do not pay attention to the topic of security. We’ve also put together a nice set of Docker security tools. I hope it will help you protect your information assets as much as possible without an impressive investment of money and time.