Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It groups application containers into logical units for easy management and discovery. As this access to compute starts to become ubiquitous, with a large market adoption, with 15 years of experience of running production workloads at Google, it leads to the discovery of an increasing amount of vulnerabilities on containers. Securing and maintaining compliance standards in these environments becomes extremely important. In this article we will focus on Kubernetes security and containers. More specifically on what kind of solution leading hedge companies are using today to face container security issues and open source tools they rely on. What are the technical best practices to avoid security vulnerabilities on container? What kind of open source tools can be used to detect vulnerabilities?
Kubernetes and container vulnerabilities
Before we delve into strategies used by companies to manage vulnerabilities, it’s worth taking a look at some well known vulnerabilities. As listed on the CVE (Common Vulnerabilities Exposures) web site, let’s explore these first:
- The Denial of Service or DoS attacks allowed users that are authorized to make change requests to the Kubernetes API Server to send a specially crafted piece of program that consumes excessive resources while processing, causing a DoS on the API Server.
- The privilege escalation flaw allowed any user who has access to gain full administrator privileges on any smallest unit of computing hardware or node being run in a Kubernetes cluster. As a result, it was possible for an attacker to escalate their privileges to gain full administrative access on any compute node running in a Kubernetes pod which are a set of node sharing the same resources and local network.
- Gaining information flow allowed a group of containers with shared storage and network (or pod) on a Node to request logs for any other pod on that Node because requests for pod log locations were not verified. A remote attacker could use this flaw to view sensitive information via pod logs that they would normally not have access to.
The previous examples are just a restricted list of discovered vulnerabilities which have been published and patched. But we need to know that in the window of time from the discovery, then the publication until the reporting and patching of a vulnerability, it can be leveraged by hackers to support several attacks like cross-container attacks during deployment or maliciously take over control of systems.
How to address vulnerabilities in enterprise?
There are several approaches that today’s businesses and IT teams can take to safeguard their organization from software vulnerabilities. Commonly, a new working methodology is adopted and security processes are automated.
Modify process to bring agility to the security world
There is a possibility to create DevSecOps team to involve security in the entire application build phase. Companies want to switch security left to developers or DevOps but at the same time there are very specific expertises like network skills or threats medication expertise that you cannot expect any developer to understand. So figuring out the organization of the entire pipeline from build phase to runtime and make sure that somebody, with excellent technical expertise, is always part of the discussion. In part DevSecOps brings out the need to invite security teams to work with DevOps teams and set plan for security process automation. It also informs developers about security visibility, feedback and known threats. DevSecOps established a common language between DevOps, developers and security teams so that they can understand themselves in terms of application.
Automation is the only way security can catch up with DevOps. Static security policies and checklists don’t scale for containers in the enterprise. The pipeline needs more security services. Automation is the way to get massive scalable container deployment and security compliance policies. To be able to automate a workflow in the pipeline where containers are network segmented from each other, leading-edge companies specify how services behave and interact during runtime so that they can decide what security policy should be applied in order to automate them.
To be able to do effective security in an organization, the company needs to automate to the point where they can create security rules automatically and then have a common vocabulary where they could talk about infrastructure as code. For example configuration has to be delivered as a piece of source code or security as code. Consider a security risk being automatically detected, reported, patched, tested, and deployed while your IT staff are asleep. Your system could self-heal, gather relevant information to discover if and where an attack came from, all without losing uptime. Automation also reduces security mean time to reaction from weeks to days to hours to minutes just as developers have reduced deployment time.
Container security best practices
If we’ve learned anything from the past couple of years, it’s that computer security flaws are inevitable. Vulnerabilities are reported, and still many organizations adopt the next product, or the next upgrade, or the next patch. “This time it’s secure,” they say. So far, it hasn’t been. The only way to effectively do business in an insecure world is to put processes in place that recognize the inherent insecurity in the products. To protect enterprise resources from attackers aiming to take advantage of vulnerabilities and to reduce the risk of exposure regardless of the products or patches, organizations should implement security best practices. Let’s go through some of them one by one:
Trust & Restrict
Always have images taken from trusted sources whether by signing the image with the digital signature or by using any private registry storage. Images should be scanned for common and known vulnerabilities (CVEs), insecure configuration, credentials leaking, insecure permissions on sensitive files, compliance with industry standards (PCI, NIST, CIS), and unsupported software licenses. Image scanning should be able to uncover vulnerabilities contained within all layers of the image, not just the base layer.
Image scanning is necessary but not enough. Container images are immutable but running containers is not. When a container is in production new vulnerabilities can be discovered, softwares can have bugs, there can be changes in data and permissions, or memory leaks. Automated tools for configuration check can scan the entire Kubernetes environment to assess its conformance with established guidelines like the CIS Benchmarks.
Inspect and Secure the Network
Network is the most critical part of your container security. Protection capabilities like firewall and end point securities should also be used to protect containers and all the East-West traffic. Kubernetes, or any other container orchestration tool, has some default security configuration and usually it is very important to go over, understand, and change it accordingly.
Reduce the attack surface to the bare minimum
Attack surface is the sum of the different point where someone an unauthorized user can enter data or extract data from your environment. Usually attack surface is smaller when speaking about container because within Docker you can choose lightweight distribution image like Alpine, designed for security, simplicity, and resource performance. With such a container, surface attack is reduced because only essential libraries are embedded in the container.
Recommended pre-deployment security steps
The previous listed security best practices is not exhaustive as it can be different in each organisation with specific requirements. But there are still some pre-deployment security steps any DevOps should observe to build efficient images, protect each Node and prevent threats like insecure configurations, credentials leaking, insecure permissions on sensitive files. Following are some general recommendations and guidelines:
- Use namespaces to isolate container and prevent privilege-escalation attack
- Restrict Linux capabilities to avoid executing processes as root user. For containers whose processes must run as the root user within the container, you can re-map this user to a less-privileged user.
- Enable SELinux, designed by NSA, it is Security-enhanced Linux (SELinux) with additional security policy over all processes.
- Utilize secure computing mode (seccomp), a Linux kernel feature used to restrict the actions available within the container.
- Configure Cgroups to limits an application to a specific set of resources
- Use a minimal Host OS, by preferring minimal images that bundle only the necessary system tools and libraries required for your project, you are also minimizing the attack surface to the bare minimum.
- Update system patches
- Run CIS Benchmark security tests
- Never put secret in Docker file
- Do not forward privileged port (22!)
- Force user directive
- Do not Copy/Paste
- Use trusted base image
- Avoid unnecessary packages
- Have reproducible immutable images to avoid having different results between runs.
- Don’t use “latest” in production. If you are using “latest” in production, you can’t rollback. There is no way of saying “latest -1”. So, it’s best to come up with meaningful tags for your images.
Open source Kubernetes security tools
While commercial tools offer multi-vector protection and visibility, there are open source projects which continue to evolve to add security features. Here are some of them to be considered for projects which are not as business critical in production.
- Falco. Falco is an open source project, hatched to understand container behavior and protect your platform from possible malicious activity. The rules engine can then detect abnormal activity in applications, containers, the underlying host, and the container platform. Falco is part of Kubernetes Response Engine architecture that allows to process security events executing playbooks to respond to security threats.
- Istio. Istio creates a service mesh for managing service communication, including routing, authentication, and encryption, but is not designed to be a security tool to detect attacks and threats.
- Anchore. Anchore is a container scanning solution that adds reporting and policy based compliance.
- Clair. Clair is an open source project for the static analysis of vulnerabilities in application containers
Security of container is becoming so critical and important on recent surveys because it is now going into production for real business services. Companies are realizing there is a whole workflow that they have to secure in their entire pipeline to avoid suffering from several threats. Kubernetes, like all softwares, is not immune to security issues. The large market adoption on containerized application sum image and container vulnerabilities to the known standard security issues and to the discovery of new threats. To address vulnerabilities, leading edge organizations need to involve agility in security process, automate security process, and implement security best practices in order to be compliant, secure deployment platform and manage security concerns at big scale.
- Fix of directory traversal on kubectl
- Photo by Philipp Katzenberger on Unsplash