There’s been a lot of discussion of container security for images, platforms, and the OS. But not much about getting visibility of the container network, especially for security purposes. Docker networking can be a complex, big topic, and Docker network security is not well understood. A common question we often hear is:
How can I see the container network behavior in real-time?”
Here are some tips for getting visibility into container networking, even with overlay and encrypted networks. This post is a summary of a recent webinar we did with our partner Cogniance. You can watch this webinar on-demand.
Top Concerns: Containers in Production
Let’s start with some concerns that companies have with running containers in productions.
- Real-time Visibility. How can I see the application and network behavior of containers?
- Constant Change. How do I maintain security when containers scale-up or down, get updated, or new apps are released?
- Transience. If a container has been attacked or initiated an attack but is no longer running, how can I get alerted and investigate?
- Workflow Mismatch. What can our security team do to avoid slowing down DevOps CI/CD processes?
- Traditional Threats. How do I detect and protect against typical threats such as DDOS or XSS on containerized applications?
Many of these concerns can be addressed with a better understanding of basic container security and networking.
Container security can be organized into these basic areas.
- Images. This includes image signing, vulnerability scanning, and content trust.
- Use image signing and verification
- Enable Content Trust for Docker
- Keep images minimal
- Run security checks as part of CI/CD
- Run-time. This includes preparing for production including SELinux and Seccomp.
- Use SELinux appropriately, not in permissive mode
- Use Seccomp to limit syscalls
- Run containers as read-only as much as possible
- Don’t put applications into default namespace
- Host and Docker Daemon. This includes protecting the host OS and container platform.
- Keep host kernel and software up to date (see recent Dirty Cow vulnerability)
- Use existing or self-written authorization plug-ins
- Configure logging properly
- Never run an insecure registry for production CI/CD pipeline
- Always use TLS if you must expose the Docker daemon via network socket
- Network. This includes overlay networks, public cloud, and encryption.
- For single node deployments, consider disabling inter-container communication (ICC) rules and then whitelist communication explicitly
- Use overlay networks to provide additional security and isolation for multi-node deployments
- Don’t rely on public cloud security rules like security groups to protect containers
- Know how to inspect unencrypted and encrypted traffic on overlay networks
- Microsegment to provide isolation of container traffic
- Use tools such as NeuVector to get a centralized and abstracted view of container communication
Docker Network Security
We’ll take a closer look at network visibility and security. Container networking is quite a big topic, so all of the issues can’t be covered in a short time.
Single node deployments of containers for production business applications are pretty rare. Most companies require multiple nodes for HA and performance. You’ll need to become familiar with these concepts in a single node deployment: bridge, iptables, inter-container communication (ICC), EXPOSE, publish, and link. You should consider disabling ICC and then whitelist communication explicitly.
The management of multi-node container networks has become a critical component. It’s gone through quite an evolution: pipework (by Jerome Petazzoni), calico, weave, flannel, GRE/VxLAN with OpenvSwitch, native overlay networks in Swarm and ‘real’ IP per container approach (Kubernetes style) and so on. Overlay networks have become the way to provide efficient management and deployment of containers.
Overlays allow you to implement desired routing scenarios, packet management algorithms and security rules on top of the existing network stack, they can be deployed and changed rapidly, allowing applications to get new network semantics without modification of host-level networks.
Security Groups / Public Cloud
In the traditional VM based public cloud environment, security was partially provided by and enforced by the cloud service through configuration of services like security groups and NACLs. However, as overlay networks become popular, much of, if not all of, the security burden shifts away from the cloud provider to you, the customer. Microservices and containers bring a dramatic increase in east-west traffic which for the most part the cloud providers are blind to.
Unencrypted and Encrypted Networks
With unencrypted overlay networks you can apply the typical traffic interception and analysis techniques that you usually use. Listen on the host interface for the traffic that you’re interested in. For example, in a swarm deployment you can listen to traffic going from one swarm node to another.
With encrypted networks inter-container packets are encapsulated into ESP, making it difficult to see. However, the traffic is still observable from within the network overlay namespace as encryption happens at a later stage. Using a tool like Wireshark it is possible to inspect encrypted traffic. You can see an example of how to do this in the webinar presented by Sergey from Cogniance.
If you can get visibility into container communication, you can set policies for microsegmenting applications to protect them. For example application A containers should not talk to B or vice versa. Within an application, an service group like Node or Redis should not initiate connections to Nginx. And containers should not initiate any connections externally or violate policy. These policies should scale even if you deploy across multiple hosts and data centers, or on a public cloud service.
We’ve really just touched the surface of Docker network security. If you’d like more technical detail you can listen to Sergey from our partner Cogniance in this docker security webinar. This is a rapidly changing landscape with frequent updates on how overlay networks work, so stay tuned for updates on actual configuration details.
NeuVector provides an application and network aware container security solution which you can easily test drive and deploy, even on running applications.