Friday, 23 September 2016
From September 13th to September 16th 2016 our operations team received alerts that our managed DNS servers came under a distributed denial of service attack or DDoS for short. These types of attacks are nothing new to us here at Thexyz and in 2015 we made quite a few improvements to protect us from DDoS which you can read about here. We also recently assisted law enforcement after an unsuccessful extortion attempt from the Armada Collective.
Thexyz will not and does not respond to extortion attempts. We have dealt with DDoS attacks in the past, and just last year made immense improvements to our infrastructure to combat DDoS attacks.
What is a DDoS attack?A Distributed Denial of Service attack is where a criminal uses a large number of computers to send requests to particular website or IP address. If they can send enough requests, it will use up all the target’s resources and the site will appear to be offline.
Even if a DDoS attack is successful in knocking a website offline, it does not lead to data being compromised or lost. It is more like being stuck in a really bad traffic jam and unable to reach your destination until the traffic clears up.
Why would someone attack Thexyz?Since November of 2015, there have been extortion demands sent to various email providers, including Fastmail, Runbox, Hushmail, Zoho and ProtonMail. By doing so they are counting on the network not being able to cope with the attack and hoping that they will just pay the ransom to prevent. In one attack, ProtonMail did pay after several days of disruptions, although this did not end the attack.
What is Thexyz doing to stop it?We have been working with our data center and upstream providers to ensure we have strong mitigation for various DDoS scenarios so we are ready to adapt. We have also notified our local police department in Toronto who have been working with international law enforcement agencies that worked on previous attacks.
Our System Operations Team started receiving alerts for our Managed DNS service which saw about 20-25k QPS on each node i.e 5X more than our normal traffic. Support staff received multiple reports from people who could not access our website or webmail site. We immediately started mitigation via Neustar and tried to bring down the QPS count by evaluating tcpdump in order to identify any unusual pattern. We also moved all our traffic via all the 16 IPs and put them under mitigation with only Port 53 as allowed as it is used for UDP/TCP. This brought the QPS count under control and the alerts started to clear up.
Connectivity issues were sporadic yet repeatable. One of the major issues that we faced with this attack was the attack vector kept changing and making the necessary changes with the mitigation filter template took time as we needed to improvise every filter in real time. Since the QPS count didn’t show drastic improvement even after Neustar dropping 960K PPS and 350 Mbps traffic, we decided to cancel the mitigation and spread the load across all our datacenters and deployed legacy mitigation at each one. This plan worked initially but had its own pitfalls - We had to quickly move back to Neustar as another attack would have put this temporary setup under jeopardy. We had our internal team review the problem and it was decided that increasing our nodes at colo will help to load balance traffic against all available nodes (old & new) after mitigation from Neustar.
What improvements have we made?We are always working to improve your experience. Everyone here at Thexyz is committed to winning your business every day and we do our best to maintain industry leading uptime and reliability. Following this DDoS attack, we are making the following improvements.
- Terminate traffic with multiple GRE tunnels, instead of just one. If this can be done, all DNS traffic need not be pointed on our DNS nodes in one DC and can be spread out to multiple locations.
- Network stack optimizations on DNS servers, to accept more packets.
- Cross check current DNS server and verify if any optimizations can be done to increase the DNS throughput.
We have secured great discounts on domains for the rest of month. Mar ch Domains Discounts Enjoy unbelievable discounts on your p...
We tested a bunch of FTP clients for iOS devices and found these to be the best. Most of them cost between $1 to $5 but we found the small p...
Over the past few weeks our system administrators have detected an unusually high amount of traffic towards WordPress login pages. We h...
With Thexyz Webmail you have the choice of using your own domain or one of our domains. Using your own domain offers more features and ben...
Thexyz Server have added the latest stable release of PHP to Thexyz Server web hosting Control Panel. You can now use the current stab...
- ▼ September 2016 (3)
- ► 2015 (34)
- ► 2014 (40)
- ► 2013 (41)
- ► 2012 (43)
- ► 2011 (69)
- ► 2010 (52)
- ► 2009 (11)