An impenetrable DevOps cluster. Configuring and Hardening Kubernetes

The most well-known tool for containerizing and automating application deployment is Docker. Known, but not the only one: it is worthy of competition with Kubernetes. Of course, it is also of some interest to attackers. How to protect Kubernetes from hacks and network attacks? This is what today’s article is about.

In the last article , I talked about how to check the current level of security of the Docker containerization system, which is very popular in the DevOps environment, and how to increase this very level. Today we will continue to study the security issues of the DevOps infrastructure, and we are targeting a container orchestration system based on Kubernetes. We will analyze the main attack vectors, study the sensational vulnerabilities, and also assemble a set of tools necessary to secure a cluster on Kubernetes. Let’s get started!

About Kubernetes in a nutshell

I think everyone who has ever encountered the DevOps world at least once has a general idea of ​​what Kubernetes is. In short, this is a fairly popular container orchestration system (although it has nothing to do with music and orchestras). What is the popularity of Kubernetes based on and why is it so good?

Kubernetes , or K8s for short , is an open source automation system for deploying, scaling and managing containerized applications. The project was founded in 2014. The original version was once developed in the bowels of Google itself for the internal needs of the corporation, but was subsequently transferred to the Cloud Native Computing Foundation open source fund , from where it got to the general public.

Initially, K8s was conceived as a manager system for managing clusters from containers. The system is dynamic, it responds to events in real time and allows you to raise a service that will simply work without dancing with a tambourine and scale on demand. Flexible, fast, cost-effective — exactly what developers need!

In Kubernetes administration practice, the concept of pods is used . Each sub is a group of containers united by a common task (for example, on the same Docker), which can be both a microservice and a massive application spaced across several parallel machines. Kubernetes is designed to solve problems with the efficient distribution of container execution across cluster nodes, depending on changes in load and the current need for services. In other words, we have a system for flexible management of the containerization infrastructure with the ability to balance the load.

Let’s list the main technical problems that can be solved using Kubernetes:

• running a large number of Docker or Rocket containers on a large number of hosts;
• monitoring the state of the run-time environment and responding to changes in a timely manner;
• management of running containers, ensuring their co-location and inter-container replication;
• scaling and balancing a large number of hosts within a cluster.

Today, K8s has become a standard for modern DevOps environments in both large companies and startups. It is actively used in cloud services like AWS , Microsoft Azure or Google Cloud .

At its core, Kubernetes is one of the strategic components of the entire DevOps process, which is why attacks on it have always been and remain relevant. Indeed, by hacking this system, a hacker will gain access to all nodes and containers running inside K8s, and this is a direct path to compromising or leaking confidential data.

The saddest thing is that the network is still full of public K8s servers that can be accessed from the Internet. How? Widely known, for example, is the wonderful Shodan service , which allows you to search for vulnerable versions of software available from the Internet and “ready to attack”. Tens of thousands of public MongoDB databases suffered from such unpatched vulnerabilities. If you’d like to see how Shodan works in practice, you can do so using the search script available on GitHub .

An overview of the main attack vectors on Kubernetes

As we found out a little earlier, K8s has a rather complex architecture and uses many components in its composition. Therefore, the vectors and types of attacks on this system also differ. So, the total attack area includes:

• Master Node — the main master server that manages the entire cluster of worker nodes (pods) and the deployment of modules on these nodes;
• Worker Node — working servers that run application containers and other Kubernetes components, such as K8s agents and proxy servers;
• Pods are the indivisible elementary unit of deployment and addressing in Kubernetes. A Pod has its own IP address and can contain one or more containers;
• Services — network services that provide data exchange within the cluster, balancing, replication, processing requests, and so on;
• System Components — key system components that are used to manage a Kubernetes cluster: API server , Kubelet and others.

K8s Security Issues

As with any complex system designed by humans, the K8s cluster infrastructure has typical security issues that system administrators often face. Below I will list the most famous of them.

• Explosion of East-West Traffic , or East-West Traffic Explosion attack. The essence of this attack is that containers can be dynamically deployed in several independent clouds, which significantly increases data exchange traffic within a logical cluster. It’s like running trains on a railroad, which is why this type of attack is called East to West. The remote location of containers can be used by attackers, for example, to implement DDoS attacks.

• Increased Attack Surface (increased attack surface). The problem is based on the fact that each container can have a different attack surface and its own unique vulnerabilities that are used by hackers for further hacking. For example, vulnerabilities for Docker or the AWS authorization system can be exploited.
• Container compromise (container compromise). The essence of the attack lies in the use of incorrect configuration (security misconfiguration) for all cluster containers, which indirectly contribute to compromise or include application vulnerabilities. Container compromises include manipulation of internal switching, process control, or file system access.
• Unauthorized connections between pods (unauthorized connections between pods within a single cluster). Compromised containers can connect to other containers on the same or different hosts to launch some kind of attack. Although L3 filtering (ACLs) is provided by network equipment according to configured rules, some unauthorized accesses can only be detected using filtering at the seventh layer of the OSI model.

Примеры критических багов

Examples of critical bugs

Any application and system software has a number of critical vulnerabilities, and K8s is no exception. The presence of such bugs endangers the entire cluster, from running containers as a whole to specific data stored inside the database on a separate pod.

According to statistics collected from the NIST Vulnerability Registry , the most critical bugs were found in 2018 — eight pieces, and four vulnerabilities were found this year. Below I will talk about the most high-profile bugs that made a lot of noise among developers who used Kubernetes.

The first vulnerability was given the number CVE-2019–5736 . It allows a malicious container to overwrite the runC executable on the host system without requiring any user interaction. As a result of such an attack, an attacker can gain root access to the host and the ability to execute arbitrary code on it. In February 2019, the exploit for CVE-2019–5736 was published on GitHub .

Another bug identified in March 2019 is CVE-2019–11246 . It allows an attacker to deliver files from a pod to the operator’s computer or modify them with a tar binary substitution using the normal kubectl cp internal command .

Another bug from last year , CVE-2018–1002105 , is related to privilege escalation. It allows an attacker to elevate privileges on the cluster and gain access to it due to a logical error in handling API calls. Security experts rated the vulnerability at 9.8 on the CVSS scale because the flaw does not require pre-authentication and is easy to exploit. Unauthorized access is opened after sending a specific request to the Kubernetes API server. All vulnerable Kubernetes builds incorrectly process a malicious request, allowing access to the backend using the TLS credentials specified in the API server settings. A few days after the problem was discovered, the PoC exploit was published on GitHub . Later released a patch, which closes this vulnerability.

Here is a common scenario for hackers to use the CVE-2018–1002105 vulnerability.

1. Determine the location in the IT landscape where K8s is running.
2. Exploitation of known vulnerabilities of cloud providers.
3. Running commands inside a container.
4. Access to the node file system.
5. Intelligence in the internal network, horizontal expansion and consolidation inside hacked systems.

Another high-profile case involving K8s took place in 2018. We are talking about hacking the account of the well-known company Tesla in the Amazon Web Services cloud. As experts later found out, credentials in one of the Kubernetes pods opened access to the AWS Tesla environment, which contained an Amazon S3 bucket. It, in turn, stored confidential data, including telemetry. However, instead of stealing this data, the hackers began to mine cryptocurrency on one of the Tesla pods.

Strengthening the security of Kubernetes

Securing Kubernetes can be conditionally represented by two practical approaches. The first approach is to set up security options (security hardening) and use the best practices (security best practices) on all key elements of the K8s infrastructure. The second approach is the use of third-party open source utilities and commercial solutions to provide monitoring, control and management of the security level.

I will start with general recommendations and tips on how to strengthen the security of Kubernetes, which any DevOps administrator can use.

1. Widespread use of TLS encryption . For each K8s infrastructure component that supports the TLS protocol, it must be enabled. This guarantees protection against traffic sniffing, forcing the identity of the server and (in the case of Mutual TLS) the identity of the client.
2. RBAC-Based Access Model and Least Possible Privilege . Role-based access control (RBAC) provides fine-grained control over the policies by which users access resources such as namespaces .
3. Third party authentication for API server . Centralization of authentication and authorization for the whole company, for example based on LDAP and Single Sign On, helps to grant and revoke rights for employees of the enterprise.
4. Encapsulating an etcd cluster behind a firewall . Since the etcd cluster stores information about the state of Kubernetes and access secrets (tokens, certificates), it is a critical component of K8s. That is why etcd should be protected separately from the rest of the cluster, preferably behind a firewall and an isolated VPC.
5. Systematic rotation of encryption keys . Regular rotation (by date and event) of security keys and certificates is one of the best practices in securing any IT system. It allows you to limit the “radius of destruction” when the access key is compromised.
6. Regular static analysis of YAML configuration files . Confidential information declared in YAML format should not be stored in the clear in pods. And sensitive configurations and access secrets (passwords) must be encrypted using tools like git-crypt . Static analysis of the YAML configuration can be used to establish a security baseline and should be done regularly.
7. Restrictions on running containers as root . Containers that run as root often have far more rights than their workloads require, which, if compromised, gives attackers even more power.
8. Mandatory Network Policies . By default, networking in Kubernetes allows all traffic between pods without any restrictions. This setting can be restricted by a network policy — NetworkPolicy .
9. Some commands for forcing security on K8s infrastructure elements . To increase the security of the network infrastructure in Kubernetes, you can use a special set of commands that I provide below.

For the API server:

// Отключение анонимного доступа 
 --anonymous-auth=false 
// Разрешение только авторизованных запросов 
  --authorizationmode AlwaysAllow
// Форсирование ограничений прав на kubelets, включая NodeRestriction на API server
  --admission-control
// Форсирование использования RBAC-модели доступа к объектам K8s
 --authorization-mode=RBAC
// Отключение небезопасных портов
 --insecure-port=0

For etcd server:

// Форсирование использования защищенного HTTPS connections к серверу etcd
 --cert-file and --key-file 
 // Включение принудительной авторизации при обращении к etcd
 --client-cert-auth=true 
 // Использование только подписанных сертификатов
 --trusted-ca-file 
 // Отключение использования автоподписанных сертификатов
  --auto-tls=false 
 // Использование только безопасных методов подключения
 --peer-client-cert-auth=true
 --peer-autotls=false and specify --peer-cert-file, --peer-key-file
 // Запуск etcd в защищенном исполнении
 --peer-trusted-ca-file
 // Форсирование авторизации с использованием сертификатов для API-сервера
 --etcd-cafile

Recommended Steps to Harden Security Before Deploying a K8s Cluster

• Make full use of namespaces.
• Force SELinux (at least in a production environment!).
• Use seccomp.
• Use cgroups.
• Use the minimum number of running hosts.
• Do not forget about installing updates and patches (kernel updates, libraries, and so on).
• Periodically run CIS Benchmark security tests to monitor the security status.

Recommended Steps to Harden Security Before Deploying a K8s Cluster

• Make full use of namespaces.
• Force SELinux (at least in a production environment!).
• Use seccomp.
• Use cgroups.
• Use the minimum number of running hosts.
• Do not forget about installing updates and patches (kernel updates, libraries, and so on).
• Periodically run CIS Benchmark security tests to monitor the security status.

Security Audit Utilities

A Kubernetes security audit for an effective due diligence should include several of the following areas:

• end host security (Host security);
• security of the master node and the Kubernetes API server (Master node Kubernetes security);
• security of Docker daemons (Docker daemon security);
• container security;
• rules for using the RBAC access model (Properly configured RBACs);
• security of stored and transmitted data (Securing data at rest and in transit).

The kube -bench scanner utility is a nimble application written in Go that checks how secure Kubernetes is deployed. To do this, kube-bench runs all the checks described in the CIS Kubernetes test . Tests are configured using YAML files, making them easy to update as specifications evolve.

Examples of using some commands of the kube-bench utility:

// Запуск утилиты в автоматическом режиме (автопоиск мастер-ноды)
# kube-bench master
// Запуск тестов безопасности CIS Kubernetes
# kube-bench node --version 1.13
// Запуск утилиты внутри контейнера
# docker run --pid=host -v /etc:/etc:ro -v /var:/var:ro -t aquasec/kube-bench:latest [master|node] --version 1.13

Another utility, Kubeaudit , is a command-line tool for auditing Kubernetes clusters for various security issues. It supports checking many options, such as running containers under the root account, storing access secrets (tokens, certificates), the ability to write to the root directory.

To install the utility on your system, use the following commands:

# go get -v github.com/Shopify/kubeaudit
# make
# make install

You can run all available tests in serial mode like this:

# kubeaudit all

To check the security for service accounts, the following command is provided:

# kubeaudit sat сервиск акунт

And this command is to test the safe use of privileges:

# kubeaudit priv

Kubesec is a Kubernetes access secret management tool with support for services such as GPG, Google Cloud KMS, and AWS KMS backends. The service evaluates how much Kubernetes resources are using security features. The user receives recommendations on how to improve the overall security of the system. You can watch live how this utility works in the demo video .

Legitimacy of images used

The legitimacy of using images to deploy containers is one of the key security criteria. But this feature is more related to Docker than Kubernetes, so I will talk about the tools used for this very briefly. You can find more information in our Docker security article .

Clair is an open source project for static vulnerability analysis in appc and docker containers . Vulnerability data is constantly imported from known sources and correlated with the indexed contents of container images to generate lists of vulnerabilities that threaten each container individually.

Another similar utility is called Dagda . It is a tool for static analysis of known vulnerabilities, searching for Trojans, viruses, malware, and other threats in Docker images and containers. In addition, Dagda is used to monitor Docker network services and run containers to detect anomalous activity. Dependency Checker is based on OWASP and Retire.js projects for security analysis of layers like Java, Python, Node.js, JS, Ruby, PHP.

Trivy is a simple and comprehensive container vulnerability scanner that supports command line. Trivy detects vulnerabilities in OS software packages (Alpine, RHEL, CentOS, and others) and application dependencies (Bundler, Composer, npm, yarn, and so on).

Operational safety

In order to ensure the safe operation of Kubernetes, a whole arsenal of special utilities was invented. One of them is called Falco . This is a cool open source scanner from Sysdig Secure for detecting intrusions and security issues on Cloud Native platforms such as Kubernetes, Mesosphere and Cloud Foundry. Falco is able to look for anomalies, atypical behavior and warns the admin through Slack, Fluentd, NATS, and other messengers.

Cilium is a service that provides filtering of network activity based on API calls in a Linux-based container infrastructure, in particular in Docker and Kubernetes. Using a new Linux kernel technology called BPF (Berkeley Packet Filter), Cilium provides a simple and efficient way to define and enforce both network and application layer security policies based on container or module ID.

Vault is a service from the well-known company HashiCorp . It allows you to store and tightly control access to tokens, passwords, certificates, encryption keys to protect secrets, and other sensitive data from the user interface, CLI line, or using HTTP API calls.

Network Security

The Calico solution provides secure networking between containers and virtual machines in the same cluster. In essence, Calico is a softswitch that creates and manages a layer 3 network by assigning a fully routable IP address to each instance. In environments where overlay is required, Calico uses IP-in-IP tunneling to allow it to work with other overlay networks.

Another useful application — Cilium is a container firewall. The program uses Linux BPF kernel technology to filter, detect, monitor and redirect data using kernel calls. Cilium helps deploy network access policies based on container ID using tag labels. In addition, the software interprets several layer 7 protocols, such as HTTP or gRPC , which allows the use of a set of REST calls.

Commercial Kubernetes Security Solutions

And now — a few words about commercial security solutions for Kubernetes in particular and the entire cloud IT infrastructure in general. As practice shows, the use of commercial Enterprise solutions is justified in terms of the convenience of managing tools from a single window, the speed of their work and more flexible scaling provided on demand. There are quite a lot of players in the Cloud Security market, we will focus only on the most famous and proven services.

Aqua Security is a service aimed at monitoring and controlling the security status for containers and cloud applications. The service provides a comprehensive solution for automating tasks such as Vulnerability Management, Runtime Protection, Secrets Management, Embedded CI/CD Scanning, Container Firewall, Compliance & Auditing. Some of the utilities used for Enterprise are available for free use on the project ‘s GitHub page .

Cavirin is another well-known service for automatically discovering and continuously evaluating a hybrid cloud environment without installing end-point agents on end hosts. Cavirin provides continuous security baseline monitoring based on frameworks and best practices including PCI, HIPAA, NIST, AWS hardening . The service also provides accurate and timely information about security updates and vulnerabilities. Cavirin’s API-based architecture integrates seamlessly with other on-premises applications as well as SaaS and cloud services.

NeuVector is a fairly well-known and well-established end-to-end security platform for all DevOps processes and, in particular, for Kubernetes security. The service includes such features as vulnerability management for images, access control, protection of processes in memory and file systems in containers, a firewall up to the seventh OSI level.

Declarative security policy enables applications to scale quickly without manual intervention. The solution provided by NeuVector is essentially a Red Hat-certified container and Docker that can be easily deployed on every host, providing firewalling, process and file system monitoring inside the container, security auditing with CIS tests, and vulnerability scanning.

Sysdig Secure is a well-known stack of solutions for end-to-end security of cloud services. The service includes sets of applications to provide such procedures as Vulnerability management, Compliance, Runtime security, Forensics, Audit. In addition to solutions positioned as Enterprise, the vendor offers high-quality open source utilities for auditing and security control. Also, for free use, everyone is provided with a Falco scanner , which we wrote about in more detail in an article about Docker security .

As you can see, very often, along with proprietary solutions, you can successfully use free open-source utilities, which enhances the overall overall effect of their use.

Additional utilities to harden Kubernetes security

I’ll tell you about a few small utilities that will be useful to you when creating the correct access matrix to the infrastructure elements of the K8s cluster.

Audit2rbac is a tool that allows you to determine what permissions are required for certain software in K8s, while it automatically generates RBAC roles.

Rbac-manager is another tool, essentially a Kubernetes operator, that makes it easy to manage RBAC roles and link them to administrator accounts.

Kube2iam is a tool that substitutes credentials from the AWS IAM access control service for all containers in the cluster.

And at the end of our material today, some interesting resources on the topic of Kubernetes security, including best practices, utility reviews, and examples of some system commands.

Kubernetes Security resource center of O’Reilly is an online version of the book Kubernetes Security Operating Kubernetes Clusters and Applications Safely by O’Reilly Media. The most complete and informative guide to planning and configuring the security of clusters based on Kubernetes. The electronic version of the book can be downloaded for free here .

Overview of Cloud Native Security is a white paper from the Kubernetus developers describing general security best practices.

The Ultimate Guide to Kubernetes Security is a good overview article describing some of the attack vectors, security issues, and typical K8s security best practices compiled by NeuVector, a developer of security solutions for the cloud.

Kubernetes Security Best-Practices is a well-structured and informative document describing Kubernetes security features that you can configure yourself. The same document is also on GitHub .

Conclusion

Today we discussed the security of the Kubernetes container orchestration system, which is very important for the entire DevOps community. Now you have an understanding of the main attack vectors, typical problems and potential risks when using this super popular open source framework, got acquainted with the general requirements for designing and deploying K8s from a security point of view, and you know several system commands to secure your cluster and sleep better at night.