By Fei Huang
There’s been a lot of recent discussions online about container security, but most have focused on image scanning or the host and OS security. The lack of enterprise security tools for containers has caused people to focus too narrowly. It’s like the old saying “when all I have is a hammer, everything looks like a nail.”
Security is often overlooked when trying to get new technologies into production, so any security focus on containers is definitely a positive thing. For example, to harden and trim down the OS, Seccomp or AppArmor types of solutions can be loaded to set security profiles between containers and the host kernel. Container images should be scanned before pushing them into the registry and the images should be digital signed at build time. And so on.
While these are all good practices for container security, no matter what defenses or security walls are put in, we can’t assume that hackers and thieves won’t find a way in. It’s very similar to the evolution of virtual machine vs physical servers. While isolation and virtualization does improve security in addition to manageability and disaster recovery, it is just not enough. Applications can have security holes and a software bug may open a backdoor at run-time. Then suddenly compromised processes appear behind the defense lines just like paratroopers.
That is why all security for defense in depth or layered security includes real-time, run time detection of threats and violations. It’s no different for containers. In many scenarios run-time security is even more critical and serious than static security because threats are happening in real-time when applications are running, not dormant or at rest.
Experienced DevOps and security professionals know that the threats in a container environment include most of the ones in a virtualized or single OS server environment. And there are additional vulnerabilities introduced by container deployments. There haven’t been widely reported container specific vulnerabilities reported yet, but here are some of the threats to be concerned about:
General Container Threats
- Application level DDOS and xss against public facing containers
- Compromised containers trying to ‘phone home’ to download malware
- Compromised containers trying to scan other internal systems to find other weakness or search for sensitive data
- Container breakout and unauthorized access across containers, hosts or data centers
- Container resource hogging, eating up CPU/Mem/Disk/IO to impact or even crash other containers
- Live patching of applications which bring in malicious processes from a hijacked DNS or other service
- Network flooding from poorly designed applications impacting other containers
Container Attacks – Examples
- SQL injection attack gaining ownership of a database container to start stealing data
- The shell-shock bash bug allowing remote attackers to execute arbitrary code inside a container
- The heart-bleed bug causing container’s memory to be leaked and analyzed
- The glibc stack-based buffer overflow caused by a man-in-the-middle attack
- A new zero-day attack on a container causing a persistent threat
If we look at a typical application life cycle, the application was built and tested in months, packaged in days, was deployed and configured in hours or minutes (thanks container technology). But it will likely be running for years. A successful shared application may even have millions of copies running over the world. A running application has a much bigger time window to be in risk of attack. A small code bug could cause a huge run-time security vulnerability for a long time (for example, SSL heartbleed). Run-time security is so important that companies have to think about it when deploying applications in production. It doesn’t matter if it is loaded in a virtual machine, physical machine or a set of small containers.
So what’s the best strategy for run-time container security? Here are 15 tips for securing containers during run-time. I’ll start with some of the preparations that I mentioned before, then get into more advanced capabilities. You can also download the checklist and guide here.
15 Tips for a Run-time Container Security Strategy
Preparing for Production
Secure the OS, ‘hardening the OS,’ trim all unnecessary modules and files, and keep up with latest security patches
Secure the container platform; For Docker, follow Docker’s best practice guide document
Prevent unauthorized access using SE Linux or AppArmor, customize and specify the security profiles
Vulnerability scan containers in all registries
Digitally sign or do integrity checks on container images
Basic Run-time Container Security
Secure the data center – firewalls/IDS/IPS/WAF/white-listing… at the gateway or entry point to reduce the chances of being attacked by traditional means
Tear down and clean up unused containers frequently, shortening the run-time window when a container could be attacked
Load application containers in read-only/non-persistent mode to reduce risks
Advanced Run-time Container Security
Isolate or segment running containers into the minimum working zone by service or application to reduce the attack surface
Monitor for attacks against containers in real-time including application layer (DDOS, xss, slowloris) threats
Monitor container behavior for violations, paying attention to any abnormal application behavior*
Block unauthorized access to containers automatically when certain that it’s abnormal*
‘Live scan’ every running container for vulnerabilities to secure the image in use, even when new containers spawn
Automate security policy to be sure protection scales automatically as containerized apps scale up and down, or across hosts/data centers
Conduct offline analysis of the security events collected to correlate events and store forensic data for containers.
*These require that you first have visibility into what is ‘normal’ application behavior, and have properly mapped it out and have created a security policy to enforce authorized behavior.
Containerization is all about faster and deeper virtualization for applications. This makes it challenging for DevOps and security people to get the visibility and security needed without slowing everything down. A run-time security solution has to have deep application awareness to fit in and keep pace.
You can spend a lot of time trying to close all the gaps. But we know that new exploits emerge constantly and there may be unknown back doors (or vulnerabilities) to our systems. So at a minimum, deploy a run-time container security monitoring solution that knows what to look for and can catch attacks early. It’s a bonus if the solution can also prevent attacks without totally destroying containers and impacting the service too.
DOWNLOAD: Run-time Container Security Guide & Checklist
You can download a helpful guide and checklist with 15 tips for run-time container security here. The guide contains additional information and tips and the checklist can be customized for your own deployment.
Let us know if you’d like to try out NeuVector on your own. It deploys just like any other container in minutes and is compatible with tools such as docker-compose, Kubernetes, Docker Swarm/UCP, Mesos, and Rancher.