By Fei Huang
This week, over 1.35 Terabits per second of traffic hit GitHub services all of a sudden. It was the most powerful distributed denial of service attack recorded to date. After only 10 minutes, GitHub had to call for help. Luckily Akamai Prolexic was able to take care of them and blocked the malicious traffic.
So what happened under the hood?
This massive DDoS attack has been identified as one type of amplification attack, called a memcached DDoS attack. Memcached is a popular caching system to speed up networks and web servers. When a query is received, the memcached server will reply with a much larger reply packet. According to some security vendors’ analysis, a small 15 bytes of request could trigger 134KB of response. Akamai has said that a 203 byte request could result in megabytes of reflected traffic. That’s a 10,000x to 50,000x amplification factor!
It is relatively easy for hackers to launch an attack by leveraging these memcached servers. Hackers simply figure out the IP address to attack, put in small fake queries, then send them to the open memcached servers. These queries will result in a response that is thousands of times the size of the traffic going back to the victim. It was reported that there are more than 90,000 to 100,000 memcached servers which are exposed to public clouds without any protection through authentication. So, the potential for this type of massive DDoS attack is that it could take down almost any server on the planet. It really depends on how many open memcached servers a hacker has at their disposal which can be used for the attack. This time, it was 1.35 Terabits per second traffic which was produced this week with GitHub as the target.
For ths specific memcached attack, the solution is obvious: try to reconfigure or disable public facing memcached servers by putting authentication and security policies around it. It also is prudent to reduce the possibility of exposing any unnecessary services as well in order to minimize the attack surface. If you have memcached servers and don’t have a proper firewall with security policies for it, here are the steps to harden memcached servers:
- Edit the /etc/memcached.conf configuration file for memcached
- Locate the -m parameter; change its value to at least 1GB
- Locate the -l parameter; change its value to 127.0.0.1 or locallhost
- Save the changes and restart memcached
Why are these damaging network attacks happening more often than before? Why is the sheer volume of each network attack continually breaking records?
At a high level, it’s because of such fast-growing and evolving technology today. Virtualization made the cloud a practical technology that scales. Containerization is disrupting the DevOps cycle, and enabling applications to become microservices which are easily distributed and automated. On the horizon, serverless computing is leading to a much more granular computing system. All these technology waves are hugely improving and helping us to get a better world.
On the other hand, the same powerful technology can be leveraged by hackers. A fully automated connected service mesh can maximize an application’s capabilities, but a malicious service or container inside it can now damage the whole distributed cluster. We see thousands of application servers like Mongo DB, Elastic Search, Memcached etc running in the public cloud, but now when a security hole is found in these public facing servers, it becomes a massive opportunity for hackers.
For business critical production services, especially public facing ones, old school vulnerability scanning techniques are not enough. Hardening the host systems is good hygiene and necessary but that won’t address this type of advanced networking attack. Network security is critical, for both internal and external networks. A hardened platform plus a multi-vector network security solution is the best way to secure modern cloud applications.
In this massive DDoS attack, Akamai Prolexic was thankfully able to mitigate it with its strong network security technology for preventing external network attacks. However there are other types of more frequent network attacks that will get through to the application level successfully and hit services which can run in containers directly. For example SlowLoris DDoS attacks, DNS attacks, reverse shells and others which, because they are not massive attacks, don’t get reported in all the news media. As always, a layered security solution is needed to protect application stacks from top to bottom.
From start to end, this network attack only took about 20 minutes. But the response times which security solutions should detect and mitigate an attack should only take minutes or even seconds. Security solutions needs to improve to become more automated if we are going to survive this massive transition to an automated, containerized world. Security needs to embedded and enabled by modern cloud technologies in order to support modern cloud applications. Security which is smart, automated and distributed could have a chance to predict attacks and enforce controls at multiple layers during run-time. This is the vision of the NeuVector multi-vector container security solution. It is designed to be an automated security solution for cloud services platforms like Kubernetes.
NeuVector can protect running containers… find out how