The First Major Vulnerability Discovered in Kubernetes, And It’s A Big One
The big news today on the eve of the start of DockerCon EMEA has not been conference related announcements but rather the disclosure of a critical security hole in Kubernetes, and by inheritance, Red Hat OpenShift. This vulnerability, CVE-2018-1002105, is so critical with a severity rating of 9.8 out of 10 that Kubernetes and OpenShift users are being urged to immediately upgrade their version to 1.10.11 at a minimum or apply the appropriate patches.
“This is a big deal. Not only can [miscreants] steal sensitive data or inject malicious code, but they can also bring down production applications and services from within an organization’s firewall,” said Ashesh Badani, VP of OpenShift at Red Hat, in this blog post.
Attacker Could Escalate Privilege to Run Arbitrary Commands
There are two primary attack vectors to exploit this vulnerability. The first involves a normal user with the pod exec/attach/portforward privileges being able to escalate privileges to become a cluster-admin. This enables the user to access any container and run arbitrary commands in an pod, for example to try to detect sensitive data. The second involves an unauthenticated user who can access the API Server to create service brokers, which could then be used to inject malicious code.
According to the Register UK in this article, “Any unauthenticated user with access to a Kubernetes environment can hit the discovery endpoint which proxies the aggregated API server (not the kube-apiserver),” explained Christopher Robinson, manager of product security assurance at Red Hat, in an email to The Register.
Ways to Mitigate and Detect Such Malicious Activity
This vulnerability is deemed so critical partly because it allows previously unauthorized users to escalate privileges and utilize connections and commands as authorized API Server users. Therefore, this activity does not show up in the Kubernetes API server logs. It is also serious because the exploit technique is fairly simple to execute.
Besides updating to a current release of Kubernetes or OpenShift, users can try to mitigate the flaw by updating the roles and removing API Extensions, as detailed in this Red Hat post in the Resolve tab.
Malicious activity targeting the Kubernetes and OpenShift system services like the API server and kubelet could be very damaging and in this case very difficult to detect. However, there are ways to mitigate the resulting potential damage to application workloads as attackers try to expand their attack and probe for other vulnerable points in their kill chain. This is where run-time container security platforms like NeuVector can monitor all containers for suspicious network, file system, and process behaviors and provide early detection.
NeuVector Kubernetes Security Platform
First and foremost, NeuVector monitors all network, file system, and process activity of Kubernetes workloads to detect and prevent malicious connections. Second, NeuVector monitors systems services such as the Kubernetes API server for suspicious connections.
All damaging attacks, especially data breaches, are comprised of a kill chain of events as the attacker infiltrates, expands, and then does their damage. Even though attackers may try to cover their tracks or mask their connections as authorized users, the attacker will inevitably make a misstep that reveals them as malicious. Using the network is the most critical attack vector for hackers, and NeuVector provides the only next generation container firewall with packet-level interrogation and enforcement to give you the best shot at detecting and preventing spread and breaches.
This means that any attempt to connect to a command and control server from an existing pod, start an unauthorized malicious container, or connect to a pod on another node/host will be detected.
Monitoring container file system and process activity enables NeuVector to detect unauthorized activity within any application workload, so even if the attacker got into a pod as an authorized user, any activity outside of normal baselines will be detected.
And finally, NeuVector by default monitors network connections of the system services such as the Kubernetes API server to detect attempts to hack into it or break out.
Won’t Vulnerability Scanning Help?
Vulnerability scanning is the first step in a layered security approach, but it can only protect against known issues at the time of the scan. There are thousands of production Kubernetes and OpenShift deployments today which were at risk of this exploit, and probably still will be for some time.
If you have a business critical application you won’t want to rely solely on scanning, RBAC, host security and other precautions implemented before production. Getting the best visibility and protection for container networking as well as container file system and process monitoring can be the difference between preventing a breach and cleaning up after one.
This Won’t Be the Last Critical Vulnerability
As we’ve always said, over the next few years we’ll be seeing a number of attacks on Kubernetes and OpenShift as the technology matures and is hardened by being in the wild. As Ashesh Badani VP of OpenShift says in his post “Kubernetes, like all software, is not immune to security issues.” Kudo’s go to Darren Shepard, Chief Architect and Co-Founder of Rancher Labs and a NeuVector partner, for finding this critical flaw.
NeuVector will continue to protect critical production workloads requiring defense in depth with the only layer 7 container firewall combined with container and Kubernetes system monitoring. We’ll continue to add detection for new attack vectors discovered and mitigate known vulnerabilities in Kubernetes and other open source software.