By Gary Duan
On June 17, 2019, security researchers at Netflix released a series of vulnerabilities they discovered in the Linux and FreeBSD kernel. By sending crafted SACK packets to the vulnerable server, attackers are able to slow down the server’s TCP stack, incur excessive resource usage, and in the worst case scenario, cause a kernel panic. The main vulnerability, CVE-2019-11477, is known as the ‘SACK Panic’ issue, while there are related vulnerabilities CVE-2019-11478 and CVE-2019-11479.
All these vulnerabilities are related to how a kernel’s TCP stack handles the Maximum Segment Size (MSS) option and SACK packets. To better understand the scope and impact of these vulnerabilities, it is worth studying these decades-old options in TCP protocols.
MSS is an option field in the TCP header. It was first introduced by RFC 879 in 1983. It allows an endpoint (client or server) of a TCP connection to indicate the maximum TCP payload size it can receive. By reducing MSS value, IP fragmentation is less likely to happen, but the TCP stream becomes less efficient because a higher percentage of bandwidth will be spent on transmitting protocol headers instead of payloads. MSS option should only appear in TCP SYN and SYN/ACK packet during the TCP handshake.
SACK is another TCP option designed to improve TCP protocol’s efficiency. As everyone knows, TCP is a reliable transmission protocol. This is achieved because the sender resends all packets that have not been ACKed. The original design of the TCP protocol is not efficient, especially in the lossy environment, because a large number of packets have to be retransmitted.
The SACK option was proposed by RFC 2018 in 1996 to address the efficiency issue. This option allows the receiver to inform the sender not only of the last sequence number (ACK) of the continuous stream, but all the segments that have been received.
While there is improved efficiency, there is also increased complexity in the TCP stack implementation. In order to support the SACK option, the TCP stack has to be able to walk through all packets that haven’t been ACKed. The defects in the implementation gives attackers a chance to launch DDoS attacks against the vulnerable servers. The Netflix’s advisory details their discoveries, patches and mitigation approaches.
Because this vulnerability exists in the kernel network stack and the attack vector is network-bound, the malicious actors can launch attacks remotely against not only physical servers, but also virtual machines and containers. The threat is serious for any company who provides their services over the Internet. A lot of services, including Netflix’s streaming service, are running in containers. It’s no surprise that the vulnerabilities are rated as critical by Netflix, and AWS released the patched AMI images immediately after the advisory went public.
NeuVector’s Automated Virtual Patch of the Sack Vulnerability
Because the NeuVector container security solution is able to inspect every packet sent and received by containerized applications, and analyze the TCP/IP layer and the application layer protocols, the symptoms of the SACK attack can be detected and stopped before they reach the application.
Here is a demo of how the NeuVector solution can detect and actively block these attacks. The mitigation works in the same way as suggested by the Netflix researchers, but without the limitations mentioned in advisory. The threat detection and attack blocking capabilities of NeuVector are automated so no configuration or manual security policy creation is required.
First we deploy a NeuVector cluster and start NeuVector’s demo application.
$ kubectl create -f neuvector.yaml $ kubectl create -f demo.yaml
Then, we start a Kali Linux pod and install hping3 in order to generate crafted packets.
$ kubectl run -it kali --image=kalilinux/kali-linux-docker [email protected]:/# apt-get update && apt-get install -y hping3
We simulate the attack by sending TCP SYN packets with very small MSS value, targeting the ‘redis’ database.
[email protected]:/# hping3 -S -p 6379 --tcp-mss 128 redis HPING redis (eth0 10.104.72.2): S set, 40 headers + 0 data bytes len=44 ip=10.104.72.2 ttl=62 DF id=0 sport=6379 flags=SA seq=0 win=28000 rtt=3.8 ms len=44 ip=10.104.72.2 ttl=62 DF id=0 sport=6379 flags=SA seq=1 win=28000 rtt=2.7 ms len=44 ip=10.104.72.2 ttl=62 DF id=0 sport=6379 flags=SA seq=2 win=28000 rtt=1.9 ms len=44 ip=10.104.72.2 ttl=62 DF id=0 sport=6379 flags=SA seq=3 win=28000 rtt=1.8 ms len=44 ip=10.104.72.2 ttl=62 DF id=0 sport=6379 flags=SA seq=4 win=28000 rtt=1.6 ms ^C --- redis hping statistic --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max = 2.8/3.3/3.9 ms
In NeuVector’s network activity graph, you can pinpoint the sender and the victim of the attack and get details of the network communications.
Next we change the redis service to “protect” mode and launch the attack again. You can see now the attacks are stopped and cannot reach the targeted service.
[email protected]:/# hping3 -S -p 6379 --tcp-mss 128 redis HPING redis (eth0 10.104.72.2): S set, 40 headers + 0 data bytes ^C --- redis hping statistic --- 8 packets transmitted, 0 packets received, 100% packet loss round-trip min/avg/max = 0.0/0.0/0.0 ms
Using the security event log page, you can investigate details about the incident. You can also download the captured packets for forensic purposes.
The NeuVector automated container security solution extends beyond threat detection as demonstrated above. NeuVector will also automatically (micro) segment all container network, process and file system activity using behavioral learning so that security policies and protection is as simple as turning on protection. Any unauthorized network connections, container processes or file system activity will be detection and can be blocked by policy.
What About Vulnerability Scanning and Image Patching?
Vulnerability scanning is also an important step to clean up your container images in the CI/CD process. However, in the majority of cases, 100% of all vulnerabilities cannot be removed from the container images and the container hosts. If they can be removed, it may take some time to find and patch them in production. Zero-day attacks can be discovered at any time while containerized applications are in production and facing Internet users, with hackers exploiting vulnerabilities before they are even discovered and publicly disclosed.
The NeuVector threat detection and prevention capability of its complete run-time security solution provides the automated virtual patch for these attacks before the official patches can be applied. This virtual patch allows users to keep running their applications on the vulnerable servers without worrying about being compromised. This gives security teams the time and protection they need to properly assess the vulnerability, scan images, and utilize the other CI/CD security tools of NeuVector during the build, ship and run phases of the pipeline.