By Gary Duan
Tesla and Jenkins have become the latest victims of data infiltration and cryptojacking. In the Tesla case, the exploits started when a Tesla Kubernetes cluster was compromised because the administrative console was not password protected. Once an attacker gains admin privilege of the Kubernetes cluster, he or she can discover all the services that are running, get into every pod to access processes, inspect files and tokens, and steal secrets managed by the Kubernetes cluster.
Not only did the attacker locate Tesla’s Amazon S3 bucket credentials, but they also started a crypto mining script within one of the pods. Several evasive measures were used by the attacker to avoid being detected.
- No well known mining software was used, avoiding detection by anti-malware software.
- The mining pool server was hidden behind the reputable CDN services using the non-standard port. (This has become the common practice for the data infiltration. The attackers have been using public services such as Dropbox and Google Drive to upload stolen data, which render the IP and domain black-list approach useless.)
- CPU usage of the crypto mining software was throttled to make it unnoticeable.
In the Jenkins case, hackers used an exploit to install malware on Jenkins servers to perform crypto mining, making over $3 million to date. Although most affected systems were personal computers, it’s a stern warning to enterprise security teams planning to run Jenkins in containerized form that constant monitoring and security is required for business critical applications.
Detecting the Exploit’s Kill Chain
The combination of a white-list model with the network and Deep Packet Inspection (DPI) technology is the only effective way to detect the exploit. NeuVector’s container security solution is designed to protect a container environment, such as those deployed with Kubernetes and Red Hat OpenShift, while running in production within a data center. Even with all the evasive techniques employed by hackers in the Tesla case, NeuVector security containers are able to detect the trace of exploits and report the incidents at multiple stages of the attack.
A misconfiguration in a system or application is the most common cause of many incidents. This is what happened in the MongoDB and ElasticSearch ransomware exploits last year, and it contributed to the exploit in the Tesla case. It is good security hygiene to follow proper security practices, such as the use of auditing and testing software to test the configurations of orchestration tools, containers, and hosts. These are available for containerized deployments in tools that automatically run the CIS Docker Bench and Kubernetes security benchmarks. However, not all potential attacks can be prevented by good hygiene.
Automating Security for Kubernetes
Most pods within a Kubernetes cluster do not need to access Internet services. By learning the network activities of all containers, we can establish a baseline of the valid network behaviors for all container services. A connection to the Internet from these pods will prominently show up as a violation in the network graph.
Although a non-standard mining server port was used, the crypto mining software needed to use standard protocols to communicate with its server. By using DPI technology to scan all network traffic, we can not only see the users which are generating these unauthorized connections, but also the crypto mining protocol used. This information gives users more confidence that something abnormal is going on in their container environment.
In addition to network behavior, system behavior can also baselined and whitelisted. Every container service will typically have a relatively stable set of processes running. A crypto mining script that starts up will be detected, as it is a major deviation from the container’s baseline.
Hacker attacks and exploits can be very stealthy in nature. In a production, run-time environment the network is the first line of defense to stop these attacks before they reach the applications. Network activities are also something cannot be hidden, as the exploit software has to communicate with the attackers to download malicious software and infiltrate sensitive information. To determine what to look for in network behavior can be challenging, and this is where DPI can help. Combining network inspection with system behavior monitoring and baselining, the NeuVector container security solution helps customers to gain some advantages in protecting their applications against the type of attacks that occurred in Tesla.