EveryDNS Under Botnet DDoS Attack

mellow marsh writes "EveryDNS, sister company to OpenDNS (which runs the PhishTank anti-phishing initiative), has been hit by a massive distributed denial-of-service attack. The attack started sometime Friday afternoon and, from all indications, was targeting Web sites that used free DNS management services provided by EveryDNS. At the height of the DDoS bombardment, EveryDNS was being hit with more than 400mbps of traffic at each of its four locations around the world. From the article: '"We were collateral damage," Ulevitch explained... Because law enforcement is involved, Ulevitch was hesitant to release details of the actual target but there are signs that some of the targets were "nefarious domains" that have since been terminated.'" OpenDNS, which makes use of EveryDNS services, was affected for a time, until they spread their authoritative DNS more broadly. The EveryDNS site is now reporting that the attack is continuing but has been mitigated and is not affecting operations.

puppy

/., like kicking a dead puppy.

COM != NET

"The EveryDNS site is now reporting that the attack is continuing but has been mitigated and is not affecting operations." O Rly. I see it reporting a chunky man with bad hair holding an @. Please change link to everydns dot NET to continue the /. DDoS.

Re:COM != NET

What parent said. The main site is http://www.everydns.net/ [everydns.net] not .com.

Another quality, editor approved Slashdot story. Great job, guys.

Affected; Irony

This really made yesterday difficult for me.

My comp sci networking class assignment was on my home server, and I use EasyDNS. Had to bus home and put it on a USB stick. Last day of class, and the end of a particularly brutal week.

correct URL

How about linking to the correct url [everydns.net]?

Does that mean

That while they attack them there'll be less spam?

Heh

The site is EveryDNS.Net [everydns.net].

I'll keep it up for Slashdot, let me just move it around a bit. :-)

-david

Re:Heh

You must be new...oh

Link To Them

Nothing helps out a site currently under a DDoS attack like being linked to on the front page of /.

This is nothing short of organized crime

Like people who kill attorneys willing to prosecute those in the mafia. If any phishers can be found, I hope they get jailed for life.

Questions?

Since I've been getting a lot of questions from folks about EveryDNS, how we've been stable and around so long, how we dealt with this DDoS and how we manage to cover our costs I am writing a response that will probably be posted here on Slashdot tomorrow or Monday to answer all these questions.

If you have questions about this or DDoS in general, feel free to ask them here and I'll make sure to cover them in my response. I'll be writing about what we've seen and what I generally do when it comes to soaking up traffic and how we handled this event in particular. (The short answer: find the smartest people you can to help you and then start taking corrective action)

Thanks!

David Ulevitch

Re:Questions?

Because law enforcement is involved, Ulevitch was hesitant to release details of the actual target but there are signs that some of the targets were "nefarious domains" that have since been terminated.

What does that mean?
Was this a 'righteous' attack on malicious websites?
Or just some intramural warfare by one nefarious group upon another?

Re:Questions?

In short, the latter. Nothing is ever righteous when it comes to DDoS. :-)

Re:Questions?

4x400mbps == 1200mbps at times.

That's less trivial to filter, especially when your upstream isn't being cooperative. In our case, which you'll read about tomorrow or Monday, we quickly were able to jump onto a network run by some folks with very very high levels of clue; nLayer operated by Richard Steenbergen. Their website is cheesy -- don't let it fool you. They are a seriously run network providing transit across the country to a bunch of other networks. Check routeviews for proof.

-david

Re:Questions?

Bless you for offering to answer questions! That sort of cooperation is indispensable if security is going to improve.

1. How did you manage the response? The one-smart-person-in-charge-who-stays-awake-the-who le-time approach? The small-team-with-independent-responsibilities model? The review-what-happened-at-shift-change model?

2. What tactics worked, and even more important, what didn't work?

3. What sort of agreements should people have in place with their upstream ISP prior to an incident?

4. How intelligent was the attack traffic? Randomized payload? Does anyone bother spoofing addresses any more?

5. Was it a guided attack or a fire and forget? In other words, did the scum make any changes to their tactics in real time as you tried corrective action?

6. What if anything can be done in the first few minutes/hours?

7. If you had to choose between capacity and filtering, which would you choose?

Real ripple effects, even from this small event.

A client (a pretty large retail chain) was using EveryDNS for forward lookups to the mail server's A record. Mail they were sending out started to bounce because receiving mail servers weren't happy when trying to validate the sending box. In once case, a vital piece of mail sent to a state taxing authority couldn't get through on a month-end calendar deadline, causing much grief. Yes, alternate communcations channels are always an option, but it wasn't immediately clear why the two mail servers in question appeared to be hating each other.

Worse, the state government box's spam filtering appliance blacklisted the retailer's server, and a third party admin had to get involved to free things up. Quite a mess.

But the real lesson? People who say that a "cyber attack" couldn't really hurt the economy are wrong, wrong, wrong. This stuff can be really disruptive, and this was a pissant little scaled-down example. No major damage, but a lot of thrashing around, untold manhours of lost productivity, and (in the case of the anecdote in question, involving just one retail company), probably some tax fines which will require much tail chasing to get waived once the the story is clearly told, assuming the state government in question is feeling sporting about it.

incompetence effects, not ripple effects

In once case, a vital piece of mail sent to a state taxing authority couldn't get through on a month-end calendar deadline, causing much grief.

Maybe a)it shouldn't be left until the deadline and b)sent via email, if it's so damn important.

And maybe you not tell clients to use a free DNS hosting service as their sole DNS provider...

They deserve the grief

In once case, a vital piece of mail sent to a state taxing authority couldn't get through on a month-end calendar deadline, causing much grief.

That grief is well deserved. E-mail is **NOT** reliable, and delivery is **NOT**, **CAN NOT** and **WILL NOT** be guaranteed. So anyone stupid enough to entrust "vital" communication to e-mail rightly deserves to have his arse whipped real good.

Myself, a month ago I missed an opportunity to collaborate on a TV miniseries. Why? Because the moron who asked me for my collaboration absolutely trusted e-mail, and it was **THE** message that bounced thanks to a network glitch, and that moron didn't think of calling me on the **PHONE**. Well, if they were stupid enough to trust e-mail like that, they probably would have made a crappy miniseries anyways.

For casual communications, there is e-mail.

For vital ones, there is registered mail, fax or phone.

"nefarious domain" is a loaded and subjective term

What is "nefarious"?

to some.. the pirate bay and allofmp3 are "nefarious domains"..

to others "www.f**Ktimewarner.com" and "walmartsucks.com" are "nefarious domains"

and to others "www.wikipedia.org" and "www.aclu.org" are "nefarious domains".

I have a lot of trouble with the idea that DDOS attacks were being carried out in (apparently successful) attempts to wipe domains off the face of the earth..

this implies the attackers had no legal standing to take those domains offline.. then they call them "nefarious" after the fact.

What's the motive?

What reason could there be for botnet owners to attack EveryDNS? I can't see that they'd gain anything from it.

Botnet? Cal it what it is!

Compromised Windows machines network.

Where are the class action suits against Microsoft for continually producing such flawed software that makes it easy to 0wn a box?

If it wasn't for 20 some years of MS indifference towards security, there wouldn't be botnets like this, being used for DDOS attacks and forwarding billions of spams a day.

Open Letter to all Trolls

You're pricks.

Nothing positive or lasting will come out of trolling (and yes: this means you anonymous asshats on /. and in usenet).

So why not be part of a winning team and stop script kiddie'ing around from your parents basement.

Sincerely,
The Rest of the Human Race.

Stupid Me...

Did anybody else read this as "Every DNS Under Botnet DDoS Attack"?

Possible Target?

After reading the updated article at Security Watch (http://securitywatch.eweek.com/exploits_and_attac ks/everydns_opendns_under_botnet_ddos_attack.html) , I'm begining to think that the target of the attack was Phish Tank itself. Why else would a hacker or hackers launch such a large scale assault on one the worlds largest free DNS providing groups if only to knock them offline for only a few hours? I think that the domains in question are just cover for the attackers attempt at taking Phish Tank offline, i.e. divert System Admins attention and resources, backdoor you way into the Phish tank server, upload botnets and virus, then push your little red button and watch years worth of work in the security field go down in a flaming death-throe. This, IMVO, to me, seems the more likely reason other than just a few kids jacking around. Why else would such a high profile target be selected by the everyday hacker if he knew the full brunt of the FBI was going to come down on him? These people that launched this assault seem to know what they wanted and went about it knowing full well what they were getting into. Also they attacked the entire DNS groups world wide system. These things seem to indicate to me that they wanted Phish Tank to go offline but just weren't able to pull it off.

Every DNS, not EasyDNS.

Hey all,

  I have to stress that it is EveryDNS that is under attack, and not EasyDNS.com [easydns.com].

That being said this is not an uncommon issue these days at DNS providers across the 'net. Before anyone starts to kick and scream about how EveryDNS is handling things, remember that these attacks can get astoundingly vicious.

No amount of "clue" or mitigation or whatnot will help when the upstream service providers themselves are having trouble with the traffic load from a large-scale botnet attack.

Re:Every DNS, not EasyDNS.

If your upstream provider can't handle 400Mbps of traffic then you're being hosted by a pretty shitty ISP/data-center. It's not like gig uplinks are expensive (even if you only commit to a tiny rate you can generally get gig uplinks). Spread this across 4 or more datacenters and you've got a lot of bandwidth.

Not to mention that networking people generally don't give a shit about bandwidth- it's packets per second that kill routers, not bandwidth. Assuming 100 byte packets that's about 4Mpps- Even a basic 7600 can handle this kind of traffic. Assuming 30 byte packets (can't be smaller than that) you're talking about 15Mpps. Again Even a basic 7600 should be able to handle that- not to mention a Juniper M7i or similar. Most Foundry equipment would laugh at that rate. All of these routers can do ACL's at full packet rates.

That said- other recent DNS attacks exceeded 1.5 Gigabits per second of traffic and were a lot more vicious than the attack being described here.

I'm not knocking EveryDNS- I know what a bitch dealing with a DDoS can be- the problem tends to be that most people aren't ready to deal with it. Using BGP community based nullrouting most service can be restored within seconds of the target IP(s) being identified. That allows admins to keep untargeted systems and services up while the attacked systems are dealt with. The admins can then use the time to locate some/any pattern in the attack or enable the appropriate filtering such as a Cisco Riverguard or similar.

-sirket

sue

sue each participating machine owner for neglegence

if you have a dog and it bites someone or damages someone's property you are liable, so why not computers?

DNSPark, too

I use DNSPark [dnspark.net], and they were subject to a DDOS attack earlier this week, too. Are they affiliated with EveryDNS too, or is it coincidence, since they are another cheap/free DNS host?

Bah My sites got hit

Had to take my home server off line as three of my sites all had time out from too many mysql connections. Notices my home cable connection was slow as hell.

At least...!

there wasn't a link to EveryDNS in the article.

The DDoS Flu Is Going Around

Two of my clients were affected by separate DDoS attacks against their hosting companies this week, and another was affected by this one. It must be contagious ... either that or I'm cursed.

Oh noes

INTERNET TERRORISM.

Thank you David!

I just want to say thank you publicly, you run a service that has helped out many folks, myself included.
And a reminder, EveryDNS.net runs on donations.
EveryDNS Donations [paypal.com]

Thank you again.

ps: Wow, slashdot uid 18.

I wish everydns had SPF support

Then it would be perfect.

Re:Poor engineering?

...400 millibits per second of traffic...

I would hope so. That would be 400/1000 bits of traffic per second. ITYM Megabits.

Re:They never anticipated Windows.

I'm no MS fan, but it is worth noting that most of the OS's you mention were even less secure than Windows at the time the internet was being developed.

The difference is that very few people knew the exploits and fewer still were in a position to actually use them.

Re:solution to DDOS attack

Not quite- It generally works like this:

First off- be prepared for a damned attack and don't wait til it happens. When an attack does come:

1- Identify the target IP address
2- Immediately null-route traffic for that address (preferably using BGP community based null-routing)
This gets the rest of your systems back up and gives you time to work on the problem.
3- Try to identify a pattern in the attacking traffic- use a product from a company like Mazu- or just tcpdump if you're good with sed and awk.
4- If there is a pattern ask the upstream ISP to block based on that pattern (same source port, same source IP, same TTL, whatever). Or block it yourself if you have the router and bandwidth capacity to deal with the attack yourself- though that's generally a waste of your resources.
5- If there is no pattern but the traffic is malformed then enabled a Cisco Riverguard or similar protection device that can filter out malformed traffic at the higher protocol layers. As an alternative, sign up for such a service form a company like Prolexic.
6- Remove your null route and see how you did.
7- If you can't afford a protection service, you can try moving the host/dns records to new IP's. Sometimes the attacks don't follow- sometimes they do. It's often worth a try as it can be done faster than enabling protection services in many cases. In this case leave the old null route in place until the attack stops. Be prepared for the attack to return at any time once they realize what's happened.

Make sure to keep traffic logs for law-enforcement and to share with other ISP's so that they can track down the offending bots.

In the future try to keep your traffic as segregated as possible such that an attack on a single host will not take down too many other services should you need to null-route that address for an extended period of time.

The easiest solution- block all IP addresses assigned to the APNIC region and watch as your site immediately returns to normal. Sadly most of the DDoS's I've seen recently had the majority of their traffic sourced from APNIC addresses.

-sirket

Re:Suck it, spammers

everydns and easydns are two seperate dns services.

Re:lkjljk

Pestiferous you say? I say gratuitous.

Re:lkjljk

automatic complaint-letter generator [pakin.org]