The Slashdot DDoS: What Happened?

What follows this introduction is a rough summary of the crazy hell that we endured with the intermittant DDoS[?] attacks we experienced last Thursday through Saturday. I'm sorry it took this long to put this together and tell you what happened, but as these things go, we were too busy trying to solve the problem to waste time talking about it. Big thanks to Andover.Net's Netops PatL, Martin and Liz, as well as Slashcode-wranglers PatG, Chris, Marc, Kurt and CowboyNeal, plus scoop (from freshmeat) and others who chimed in along the way. Tomorrow is part2: A good description of how the new Slashdot @ Exodus works.

What follows is more-or-less Pat "BSD-Pat" Lynch's account of the DDoS... Pat is our super 31337 BSD Junkie sysadmin. He wants everyone to know that the timeline below is little screwy, but things are more or less in sequential order. Things might not be exactly perfect, but hey, what do you expect after 30 hours without sleep?

Having moved the day before, none of us were truly familiar with exactly how the new hardware would handle the full burden of being 'slashdot.org'. The cluster (known affectionately as The Matrix) had handled its premiere day with flying colors, but we didn't really have an accurate feel of how things would react. Combine this with a couple of extremely high traffic stories posted on both Thursday and Friday, and it took us a awhile to determine that the problems were external, and not a flaw in some new component in the cluster."

The Attacks began Thursday morning. Most of it came in the form of SYN floods, from obvious /16's no less, and some /24's. We didn't have any zombie-killing software or a firewall installed because of certain network topology issues. Later on, a second wave came, this closer to 8 or 9pm and the load balancer (an arrowpoint CS-100) died under the load.

The DDoS, as far as I could see, was a lot of SYN and Zero port packets coming from various /16's and /24's as well as a bunch of RFC1918 reserved addresses (10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16) At one point we reached 109Mbits worth of traffic into our network.

Liz and I went back to Exodus and rebooted the Arrowpoint, then the site seemed "ok" for a bit. By 3 in the morning, Liz decided that the PIX (Cisco's firewall) could simply not do what it was supposed to do, so we went back and started building a FreeBSD box as a bridging firewall.

just before we went to plug it in, I tried to ssh into the vpn-gate and noticed that nothing was working right: while the site worked, outgoing traffic and source groups on the Arrowpoint was screwed. As if that wasn't enough, two ports died on it already!

At some unknown point (time blurs after 30 hours straight!) Martin and PatG show up (thank the gods!) and they force us to go to sleep, they bring the site up outside the Arrowpoint, while Liz and I watch from a hotel room.

As of Friday morning, the site is semi-working, but the adsystem can't be updated, and we have no access to the backend servers. I scream bloody murder to Arrowpoint, who eventually shows up to blame the router: a cisco 6509 switch with two RSM/MSFCs.

Liz and I do packet dumps and determine it's not the router, the little CS-100 had died the night before, and thats where it all started. The Arrowpoint guy insists we did something to make the Arrowpoint not work (CT: Explicit description of precisely where Liz and and Pat wanted to store the newly deceased Arrowpoint removed to keep things rated PG) By 7 the CS-800 CSS is up we're almost done for the day, but we stay to make sure. By 10pm we're exhausted but stable, although we're running 4 servers on a round-robin DNS while the new load balancer waits.

Netops (Liz , Martin and I) regroup, and do reintegration of new Arrowpoint CS-800 and installation of a new FreeBSD Firewall box instead of the PIX during Saturday Afternoon. Slashdot returns to normal. Sysadmins get well-deserved sleep.

So that was the story. It was a pretty hellish weekend for everyone involved, but thanks again to those that helped get our ducks back in a row. Again, Part #2 to this (which originally was gonna be run last Thursday, but with all this ddos stuff got pushed aside) is a fairly detailed description of the new Slashdot setup at Exodus, complete with all the changes mentioned above. Fun for the whole family if your family is really into clusters of web servers."

Defense in Depth

[Old Man Kensey]

Modern military command uses the concept of defense in depth. The essence of this is trading space for time.

The simplest case is building two small walls instead of one humongous wall. If you build a humongous wall, it takes a long time to get through... unless the enemy finds a single weak point -- then you're screwed. Two walls each take less time to get through, but if they're well-built using different techniques, the enemy may not get through to begin with and if they breach the first they lose time covering ground and then adapting. They're also very obvious as they traverse the open ground between barriers.

Network security can benefit from the same concept. Others have already mentioned heterogeneous "airgap" systems -- one of the most common and least excusable faux pas by so-called "security admins" is a single firewall protecting a herd of boxen. Second to that is identical airgap firewalls.

Of course real defense doesn't end with the walls. Even services running behind an airgap should be structured with an eye toward reasonable security, as others have pointed out. Many companies think their firewalls make them safe; come the day those firewalls are breached and the attackers make off with everything stored on the NT intranet server before wiping the drive, they'll find out differently.

Any server, no matter how well shielded, should start life in a lockdown configuration and then be made less secure only as needed ("do we really need to enable daytime on this box?"). Admittedly I haven't kept up with developments in secure distros, but does anyone make a "locked-down by default" distro based off Red Hat/Debian/*BSD? It'd be a real service to admins and if not it's something I might consider starting a project for. I know of Bastille Linux but that's (as far as I know) not so much a distro as a set of scripts to tighten up Red Hat.

The only thing we have yet to figure out is how to effectively make systems under attack "shoot back". The most they can do at the moment is call in an airstrike (i.e. alert the admins). Any return-fire capability would only be as good as the intermediate links let it be. It might not even be a good idea, as it would increase network traffic and make the attack that much more severe.

Re:Uhh, not quite

[JamesSharman]

OK, ok, everyone point fun at me for being dyslexic! And anyway, how do you know they weren't falling over chickens!

Ok, so I'm dyslexic and get the spelling of words wrong sometimes (which a spell check helps with) and sometimes use the wrong word (which it doesn't) but there is a deeper issue here. Language is simply a means of communication, if the message is communicated then it has done it's job. Furthermore language is not defined by text books and dictionaries, these books record it. There is only something wrong with a statement when it fails to convey the intended message, not when a word is incorrectly spelt or a comma is out of place.

Referring to me as 'an ignorant looser' is nothing short of bigotry, if you really want a discussion I suggestion come out from behind the aptly named 'anonymous coward' hiding place, reveal your identity and discuss the matter without resorting to insult.

Re:What's the Cisco angle?

[buffy]

"They are probably much better off with the BSD box. Although it's not a good idea to advertise their security infrastructure layout to the world. (Hint, Hint, CmdrTaco!)"

I disagree 100%. Knowledge of an installation's infrastructure should never comprimise the security of the setup. If it does, then you're relying (to a certain extent) on security through obscurity. Security should be provided by a well thought out layered approach: network layering (multiple firewalls, screening routers, IDS, etc...), host-based security (tcp wrappers, service minimalization & replacement, tripwire, etc..), and application security (ie. authentication, verification, etc...)

In designing networking/server infrastructures it's best to think of it as an open source project, and you should be willing to get opinions and discussion from any number of sources that could include crackers who may at some point want to use that knowledge to attack your site. This is one of the things I like about TIS Gauntlet once upon a time..."crystal box" was the term they used to describe it.

You should prepare for an attack ASSUMING that the infiltrators know as much about your setup as you do. In the long run, if you know that your infrastructure can hold up to someone with that amount of knowledge, then you'll be doing pretty well.

My only question...did I actually see in a comment that they're using NFS to publish data to the distributed webservers??? Ew. Run.

-buffy

(Hmm...I seem to really like parentheticals, don't I? (well maybe not. (really!)))

Re:Why a firewall?

[ryanr]

Probably because they're running things behind the firewall like NFS and some flavor of SQL which won't be secure enough to expose to the Internet anytime soon.

Re:But who ... ?

[G27 Radio]

Who knows, even Bill may be a /. reader?!

I'd be suprised if he wasn't. I just wonder if he posts.

numb

Re:DDOS != 10.0.0.0

[InsaneGeek]

Not knowing if that's a joke or not...

I may sound like too much of a bastard, but not having time is not an excuse, you aren't doing your job. Each of those routers had to be configured to begin with, and most networking guys keep the entire configs in a text file that they upload to a router, add a couple of lines to the code and your done. Not doing this stuff is akin to doing a ("chmod -R 777 /") for all of your unix boxes because it takes time to setup accounts, etc.

It's amazing how much time the admin seems to get when a site realizes that 80% of a T3 is full of bad traffic (the old saying and ounce of prevention...). If you don't have time to do this type of stuff, you need to have a serious talk with your boss; because sometime soon you are going to spend a whole week cleaning up some crap that would have only taken you a couple of hours to do in the first place (not to mention boss yelling, legal dept. yelling, ceo yelling, customers yelling...).

Re:A little more detail on the hardware setup

[Tet]

There are currently six VA FullOns serving web pages from an NFS server

This is something I've been wondering about recently. How do you have clustered web servers sharing storage? Sure, use NFS you say. But that introduces a single point of failure. If your NFS server goes down, you lose the entire cluster. Are there any solutions to this that don't involve spending vast quantities of money on a Sun HA failover system or an Auspex mirrored NFS system or similar?

Re:10.0.0.0 net

[snopes]

This is a very misleading post. First of, it's 10.0.0.0/8 not 10.0.0.0/16. Second, the only net you could remotely finger is the originating net for not doing egress filtering on the private nets. Everyone else is just routing based on dest IP and switching based on the data link (MAC) info. But there's no requirement for them to be doing that. The real fault lies with the local network engineer for not doing ingress filtering of packets with a source on a private net. You've got to take responsibility for your own misconfigurations. You can't blame everything on somebody else. They should have had a firewall in place and Exodus should have been doing the ingress filtering at their border. See my other post [slashdot.org] for a suggestion as to why this wasn't happening.

Re:GODDAMIT A LINUX SITE SHOULD RUN LINUX!

[Darchmare]

---
OSX is covered cause even tho apple is a hugely proprietry company, everyone here loves microsoft competitors.
---

Well, Apple was a Microsoft competitor long before Slashdot started seriously covering them. I think it's more the hardware and Unix-based nature of their recent OS movements more than anything. Note that OSX is based on Mach/BSD, which goes to show you that they're not focused on Linux only.

---
Anyway, point was slashdot IS primarily a linux site.
---

If by that you mean that most of the people here have an interest in it, sure.

I'm not saying that Slashdot isn't incredibly biased toward Linux, but that doesn't mean the Slashdot editors won't use *BSD when the occasion warrants.

Anyhow, Slashdot may be Linux-oriented, but nowhere do they say that they are so to the exclusion of everything else (which was the point I was arguing).


- Jeff A. Campbell
- VelociNews (http://www.velocinews.com [velocinews.com])

Re:Blame Canada

[Kurt Gray]

Exodus is getting $1million/year from us so they let us do whatever we want. They only thing they won't let us do is take a picture of our cage -- no cameras allowed anywhere in the facility! I guess they're afraid we're going to steal their soul. We were able to smuggle out this picture [fourmilab.ch] of PatG, PatL, Martin, and the Arrowpoint rep. Behind them you can see the current Slashdot setup.

Of course it was the router!

[Denor]

...Arrowpoint, who eventually shows up to blame the router: a cisco 6509 switch with two RSM/MSFCs.

Yeah, that Microsoft certification, you really should have known better....

Re:Tracking DDOS or even DOS is difficult.

[hburch]

At this point, it's impossible. Due to the relative statelessness of the Internet (a design feature that is required for most of its scalability), there are not 'logs' to look through that would give you the answer. The packets that Slashdot received, if they were logged, aren't going to have any information about where the attack came from (or, at least, if the script kiddie was even half decent).

There are a variety of ways to trace DoS attacks back using the current infrastructure, including the 'manual traceback' technique that Christopher alluded to. However, they don't work very well for DDoS.

For DDoS, tracing back to he source still isn't good enough, as 'here's a list of 10,000 hosts that have been co-opted to do a DDoS' has made the problem simpler, but still pretty difficult (stopping those hosts from doing it again, making sure that a different set of 10,000 hosts are co-opeted, determining who co-opted the hosts in the original place, etc.). Also, I'm not convinced of Savage's trick with chunking working very well when you're talking about 10,000 traces.

Why a firewall?

[Russ Nelson]

Why are you installing a Unix-based firewall in front of some Unix-based public servers? Why not secure the servers in the first place?
-russ

Blame Univac

[zorgon]

Hahahahahaah! You Andover folks are more 1337 than I thought. Not only do you have uber-hacker John Walker [fourmilab.ch] on your team, you're running the site on a Univac 1107 [fourmilab.ch] -- say, you have any of those old 2 1/4 ton 100MB hard disks? [fourmilab.ch]

Re:Blame Canada

[cybrthng]

You should have gone with Digex for your webhosting machines. They have a secure network, secure facilities and all the hardware/software/monitoring in place and managed as a service.

Exodus is full of people that don't pay or want the world for nothing. Digex on the other hand has great support, strict security and very effecient services. (No restrictions on bandwitdth, backup hardware/routers/nics/switches and enough power to last weeks after a nuclear strike :)

Go digex

Re:Why a firewall?

[GC]

Hmmm,

The CPU of a server spends time on a packet before it has checked what service it is destined for.
Filtering Broadcast packets on a network is a great way to improve workstation & server performance.

In the old days with Token Ring networks with cheap and nasty MAUs you could bring down the entire network with broadcast packets...

Re:How does a DDoS attack screw up the / code???

[gothic]

Because what you don't see when you come to this site, and possibly look at the code, is that all the pages are dynamicly generated. I can't be sure, but I'm guessing the 'sections' on the front page could be located on more then one server. And the artciles are in a DB on another server, so if that can't be reached, you just seriously chopped down the size of your resulting HTML. (And output, since there is no longer a middle) =]
The webpage where I work is located in 5 different files (PHP), and joined together when the user loads. But when it's all together, and you look at the source, the page looks like it should be one file.

Re:RSM/MSFC definations

[GC]

MSFC

On Cisco equipment this is touted as Netflow and on Cat5500s the feature resides on the Supervisor Module (III) - This is in Slot 1.

RSM

Essentially a Cisco series 7000 router, without any physical ports (except the backplane I suppose), like others have pointed out to route between VLANs within the VTP Domain of Cisco switches.

I have found that routing is pretty slow (as least for today's 1gb/100Mbit LANs) so you're probably better putting Gigabit cards on your servers that are capable of VLAN detagging, then just let the switch switch packets by physical address.

Re:10.0.0.0 net

[austinij]

A lot of corporations will use the 10.* address space for thier internal networks... with routers and switches and the such. Most routers will not allow 10.* addresses to be routed unless specifically told to. Unfortunately, the problem occours when the dweebs at the upstream connection point (to the net) tell thier routers to go ahead and route 10.* address "since the rest of thier network does." This is just silly and very unresponsible. There is no problem using non-routables, however it must be done correctly!

Mandrake - Secure level 5

[Kris Warkentin]

Try installing Mandrake with default security level of 5. It shuts down just about everything you can think of. Any services you want, you have to explicitly turn them on.

Re:10.0.0.0 net

[lanner]

Global Crossing / Global Center filters out all RFC 1819 (or it is 1918?, whatever) private networks on our core routers, as well as customer connected peripery routers. This includes DSL, DS1, DS3, OC, ISDN, and dialup customers.

Customers with BGP sessions are allowed to advertise these networks either.

Re:No doubt it was MS/MPAA/RIAA/Metallica/Dr Dre

[Kryptonomic]

Let's just follow Pink's example:

So ya thought ya might like to go to the show.
To feel the warm thrill of confusion, that special geek glow.
got me some bad news for you, Sunshine.
Roblimo isn't well, he stayed back at the hotel,
And he sent us along as a surrogate hand.
We're gonna find out where you fans really stand.

Are there any MCSEs on the slashdot tonight?
Get 'em up against the wall. -- 'Gainst the wall!
And that one with all the karma, he don't look right to me.
Get him up against the wall. -- 'Gainst the wall!
And that one is in RIAA, and that one's in MPAA.
Who let all this riffraff to have their say?
There's one smoking a joint, and another with spots!
If I had my way I'd have all of ya shot.

(I guess Pink Floyd's going to sue me now)

Re:Owned? - Nope

[Sharkyfour]

Make life even easier for people and point that NameZero domain to http://oneilli.net/~sharky/entry/?slashd ot [oneilli.net]... It'll break away that damn annoying ad banner frame automagically... :-) (and it gives users a choice to keep the frame so NameZero can't get *too* mad before defualting to break it off after 10 seconds...)
--

Re:Troll filters?

[Kurt Gray]

Just some posting lameness filters were added to discourage the lame trolls who post in all caps and crap like that. I personally enjoy reading the well crafted rants from the creative trolls who try to start flame wars -- I'm sorry but someone posted this one troll yesterday that the web should be a place for marketers to sell and the techie elitists should get lost -- now stuff like that is funny and I thank the troll who posted because I laughed my ass off.

Re:How does a DDoS attack screw up the / code???

[Dr. Sp0ng]

It didn't screw up the code, it most likely blasted the hell out of the MySQL servers, and the code doesn't do a whole hell of a lot of error checking.
--

Re:Of course it was the router!

[kwsNI]

Huh, that's funny. Those are very hard to play without moving the mouse. As everyone who's used NT knows, you have to reboot the system every time you move the mouse so that the changes take effect.

kwsNI

Re:Why a firewall?

[orabidoo]

and also: because their firewall is a Bridging Firewall. As far as I know bridging firewall support doesn't exist on Linux, outside of some very alpha patches (it may have been integrated in 2.3.x, which is just as alpha anyway). bridging firewalls are neat, they filter and forward stuff without having to make extra subnets for routing. the right sort of thing to put in front of a load balancer.

Re:Why a firewall?

[Russ Nelson]

You have N servers plus one firewall. All told, N+1 hosts have the horsepower to deal with the traffic. You just agreed to that, right? So why is life any easier just because one of the machines is configured as a firewall?
-russ

UNIVAC 1107

[Animats]

I spent many hours sitting at that 1107 console. You couldn't do much from there except watch the batch jobs go by. 0.25 MIPS, 256K, and it supported a whole engineering school.

John Nagle

Slashdot kind of deserves this

[consumer]

In a way, I think Slashdot is getting what it deserves. This is the site where the general consensus among posters has been that it's okay to DDoS a site if you don't like something they did. (Remember all the scripts people posted to attack eToys?) Maybe some troll got tired of being moderated down and took the other posters' advice. Or maybe RTMark decided Slashdot is immoral and staged a "sit-in". "Do unto others..."

A LOT of detail on the hardware setup

[eshaft]

So when does Kurt get his own weekely column, "Inside Slashdot?"

:)

Poetic Justice?

[JamesSharman]

I find it quite amusing that the site that has entered legend for it's own specialized form the of the DDOS (the slashdot effect) has itself fallen fowl of the more malicious variety.

Congratulations on getting the new servers up and running, I've just moved my badtech cartoon site to digital nation (The old location of the slashdot servers).

Re:Of course it was the router!

[Vanders]

Sure you're not thinking of a MCSE? Minesweeper Consultant and Solitair Expert....

Thanks for a decent explanation

[spiralx]

Well, thanks to the /. crew for finally getting round to telling us what happened - so much for all the whiners who insist that CmdrTaco et al. are involved in some massive conspiracy to keep us in the dark about "important issues" :)

Any possibility on finding out more about the origin of the DDoS? I'm not really sure of the feasibility of doing anything myself.

Could someone point me to a networking tutorial?

[gatzke]

Could someone point me to a decent networking tutorial on the web?

I use systems, and I understand IP (a bit). I do not understand the stuff between the nodes. Switches. Routers. Hubs. Firewalls. Addressing.

Most people don't have to deal with this crap casue a network guy sets it up and we plug in and use the IP address he gives us, but if I ever want to set up my own network (beowulf lab or home network) I need some more info.

I have also heard that you can directly connect two NICs with a special cable. Do you need software changes to do this?

Sorry I am so clueless.

ed

Re:Why a firewall?

[Russ Nelson]

the only thing that allows your other boxes to work at all, since anything you want to do as a webhost is inherently insecure.

Exactly my point. You're exposing your weakest service. How does a firewall increase your security when you're giving away the farm? And as Slashdot proved, it's a single point of failure.
-russ

Re:Blame Exodus SQUARED

[zrk]

BEGIN rant

I would definitely look at Exodus for some of this trouble. At times, they have been less than helpful for the service level they claim they will provide.

-They changed their security policy a while ago, and neglected to tell us until after the fact. All visitors to your cage must be announced, and just try to get replacement parts in and out without a whole rigamarole. Previously, one person "on the list" could escort others in and out of the facility, but no more. Granted this makes some sense, but when we showed up the first time after they changed their policy, before informing us, we balked, and complained. The response was (I kid you not) - "Well, we're a big company now, so we can't give the same level of service we used to." WHAT KIND OF ORGANIZATION SHOOTS THEMSELVES IN THE FOOT LIKE THAT?

-Their HVAC is substandard, and they don't truly care what equipment is placed in a cage. I pity the poor sun techs who have to replace the Sun server at the bottom of a stack of 10 other machines (ie, no shelf).

-They continue to abide by their own notification procedures when their "monitoring" software reports trouble. We've gone over their policy several times with them, and verified they had correct contact information for us, and yet they still follow old ways of notification. In this case, it's paging one person instead of using the paging mechanism that contacts the actual people who will do the work - the effort is the same either way.

-The number of times that we've notified them of trouble before their monitors catch it - for example, try working with them to show DNS requests from the outside to their servers aren't being handled.

END rant

I could go on, but I won't.

Nice account, but who?

[alteridem]

That was a good account of what happened, but in part two, we want to hear what you are doing to track the bastards down. Knowing how you go about fixing the problem and then tracking down the culprits may help other people who run into the same problem in the future. We would understand if you need to keep the info secret until you have finished tracking them down, or for legal reasons, but at least tell us so.

Re:Exodus? BAH!

[Kintanon]

Or until someone sniffs their router password and blows away their routing configuration....


If by sniff you mean write down while working there. That was an ex-employee, disgruntled and whatnot, that had access to the information. Not a technical exploit, but a social one.

Kintanon

Re:Why not a firewall.

[warpeightbot]

2) A firewall breaks the end-to-end communication paradigm of the Internet. The idea is that you place smarts in the middle. Sorry, no. Hosts should communicate with hosts, not with intermediaries.

This is fine in a world that has never seen the Internet Worm, much less a real DDOS attack. Back when sysadmins were treated as gods and deserved that honor, when spaf@gatech spoke and it Was... But no, now we have dime-a-dozen dialup accounts, and every baggy-trousered pre-high-school geek with a dusty 486 in a corner and $15 can be Master of his own do.main (and cares not a fig for authority, but that's a whole 'nother can of worms). In short, it's wartime in cyberspace, and a gentleman's handshake is worth the paper it's written on.

2) A firewall breaks the end-to-end communication paradigm of the Internet. The idea is that you place smarts in the middle. Sorry, no. Hosts should communicate with hosts, not with intermediaries.

Nice theory, but the poster hasn't been reading the thread. It's much easier to secure a host when all it has to do is flip packets. You can also (theoretically) (I know it's in Linux 2.2, I just haven't heard of it being put into practice) imbed some QoS/load balancing smarts into a proper bridging router box... and since the kernel brings the interface for this out into userspace, you can write whatever kind of balance/filter algorithm you want. (dunno if OpenBSD does this by default, but, after all, what's Open Source for? :) And to label one of your servers the "firewall" and put the load balancing softwre there rather than in front of the server pool kinda defeats the purpose of load balancing, no?

So maybe he was a troll, but it's an obscure enough subject that somebody would take him for real.... and if I end up giving somebody somewhere the real picture, then I'll have done what I wanted to do.

--
Use the Force
Read the Source

/. hardware

[josepha48]

I'd be curious ot know what slashdot.org is running. In hte ways of hardware and software? I heard they have mysql, and now a freebsd box but what else? This information would probably be interesting to anyone else out there that has an internet site. IE how does /. perform and what do they use to get their performace?

send flames > /dev/null

Re:Why a firewall?

[Christopher Thomas]

Why are you installing a Unix-based firewall in front of some Unix-based public servers? Why not secure the servers in the first place?

Having a firewall in place to filter invalid packets and other crud thrown at the servers means that more of the servers' time is spent generating slashdot pages. Also, the simpler the Unix box, the easier it is to secure - hence, securing a stripped down firewall instead of a big, complex slashdot server.

Re:Defense in Depth

[studerby]

The only thing we have yet to figure out is how to effectively make systems under attack "shoot back". The most they can do at the moment is call in an airstrike (i.e. alert the admins). Any return-fire capability would only be as good as the intermediate links let it be. It might not even be a good idea, as it would increase network traffic and make the attack that much more severe.

Since most attacks are staged through innocent 3rd parties, auto-reprisals are likely to make the damage worse. And if someone effectively spoofs the reprisal software, they could use *your* defenses to stage an attack on someone else. Can you say "liability lawsuit"? I knew you could...

Re:Defense in Depth

[klevin]

> Admittedly I haven't kept up with developments in secure distros, but does anyone make a
> "locked-down by default" distro based off Red Hat/Debian/*BSD?

In the Linux area, take a look at the Nexus [nexus-project.org] project. It's being built from scratch, as opposed to being based on an existing distribution.

http://nexus-project.org/

MPAA

[austinij]

(CT: Explicit description of precisely where Liz and and Pat wanted to store the newly deceased Arrowpoint removed to keep things rated PG)

You'd better watch it with this comment... the MPAA might come after you too!

Timing

[348]

I'm curious about the timing with the port to the Exodus environment, was there any indication the attack was timed to take advantage of the different environment? Not saying that the security measures were better or worse than the old site, just that the timing seems rather convienent.

Re:Twas an inside job, my friend

[Rendus]

Sengan violated the biggest rule of Slashdot in that everyone can state their opinion on something. That wasn't nearly as bad as how he did it though.

He posted a flaimbait story, and disabled comment posting (the only story to EVER have this done that I know of, and I've been around since quite nearly the beginning of slashdot. Remember TCWWW anyone?), put his flaimbait opinion on it, posted some horribly incorrect information, and expected people to be happy with him about it. He marked it as a news piece, when it was more editorial than anything.

That's why you rarely see Sengan around anymore. After that he was constantly flamed on every story he posted (I think he continued to post for a little while longer).

Re:security by obscurity

[noom]

As we all know, security by obscurity is no security at all.

I really don't see why that should be the case (in fact, it's obviously false), but, considering that the statement rhymes and uses alliteration nicely, I can see why someone might be convinced. Don't you hate it when you can't get stupid jingles like that out of your head? Especially that damned "Mmmm Bop" song...

-NooM

What's the Cisco angle?

[drteknikal]

I'm curious on one detail. What was it that the Cisco PIX was supposed to do and didn't?

Re:Why a firewall?

[PigleT]

OK, so how do you "secure" e.g. mysqld? You can restrict the host access down to "localhost" so that only locally-running CGIs can access it and you can put usernames and passwords on it so the port isn't wide-open at a 'logical/application' level, but even so, if someone gets into the one machine, they've still got the whole lot and they can examine your CGIs (`man strings` and slashdot uses perl, too!) and so on.

IOW, you can't secure mysqld properly if it's all on the same box.
Apply the same for Oracle and just about everything else you want to run. There's a maximum level of security attainable by any one of these things (I won't say they're all "un-securable", merely "of limited securability"), and the closer to the box on which the service runs the cracker gets, the higher the risk.

So you put things on two boxes, yeah?
~Tim
--
.|` Clouds cross the black moonlight,

Re:Why are packets from reserved addrs geting thru

[horape]

that's not true. for example, if you traceroute an erols dialup, you'll see the 10. for the ppp server, and no that doesn't break the rfc. iirc, the rule is you can route from 10., etc but not from (eg, icmp from a 10. is ok, but if you try to telnet to a 10. it shouldn't go past your local border).

Nope, these 10.0.0.0/8 address you see in the traceroute are badly configured machines. For example, it breaks MTU path discovery very often...

Re:Why a firewall?

[Fas Attarac]

I don't quite know if you're asking these questions because you're legitimately trying to learn something about security or if you just think you have all the answers and are considering the universities that teach this stuff and the highly trained corporate IT departments to be idiots..

I totally agree that systems need to be individually secured against obvious problems. In any production setting you have to safeguard at least a bit against unauthorized access (even if from your own network). Firewalls just allow that to be done in a single layer, with a single access policy and set of rules.

It's a lot easier to set up a firewall (perhaps composed of multiple systems for redundancy and load management, perhaps even built into the very routers you're using) that's been designed for this task than it is to go through and audit every system individually.

What if you don't want systems to be reachable from the outside world at all? Your solution would be to use ipchains/whatever and just block all of the ports with that?

Are you aware that there are regularly discovered stack flaws that allow people to disable or crash a system where they have a direct network path like this? What about OS fingerprinting? I would be very uncomfortable if my servers could be touched at all by packets originating from the Internet. Firewalls not only keep people from accessing what may be potentially insecure systems, but it keeps them from doing *anything at all* to them that isn't explicitely allowed. By putting this functionality into a firewall, you have only one type of system (by "type" I mean "firewall" versus "web server" versus "NFS server" or "database server") seen by the outside world, and no critically vulnerable services that they can even *see* much less get to. If you were to put the load of network security onto the individual hosts, there are tons of things somebody can do, even if the service itself is secure, network threats are still quite serious.

If you legitimately are curious about actual network setups and why things like firewalls are necessary and aren't just trying to be an ignorant troll, I'd suggest you take some networking classes at your local university. Depending on their setup, they may have a lab for people to play around with various types of setups, even to the point of letting you simulate your own DDoS attacks and hacking into your own systems. Fun stuff.

Owned?

[Remote]

I wasn't going to talk about this in public because of /. silence about the DDoS, for I thought things could be somewhat related.

This is what I got this morning when I asked for www.slashdot.org:

&lthtml&gt
&lthead&gt
&lttitle&gtNot Slashdot.org&lt/title&gt
&ltmeta name="keywords" content=""&gt
&ltmeta name="description" content=""&gt
&lt/head&gt

&ltscript language="javascript"&gt
&lt!--
if (top.frames.length != 0)
{
top.location=document.location
}
//--&gt
&lt/script&gt

&ltframeset
rows="*,90" marginwidth="0" marginheight="0"
framespacing=0 frameborder=no border=0
&gt
&ltframe
marginwidth="5" marginheight="2"
src="http://slashdot.org"
name=thepage framespacing=0 frameborder=no border=0
&gt
&ltframe
marginwidth="0" marginheight="0"

src="http://red.namezero.com/strip2/strip.jhtml? name=slahsdot.org&channel=www"
name=pb scrollbars=no scrolling=no
framespacing=0 frameborder=no border=0
&gt
&lt/frameset&gt

&ltnoframes&gt
Sorry
&lt/noframes&gt

&lt/html&gt

Weird. Did anybody else see this?

Blame Exodus

[snopes]

We didn't have any zombie-killing software or a firewall installed because of certain network topology issues.

Topology my ass. Exodus fights hard to make you use their 'value add' security services. Be honest guys, the reason you weren't protected was b/c those bastards were working you over for more money and don't want you running your own security, right? In fairness, there's some nice things about running out of an Exodus facility, but dealing with their physical and network security chimps is not one of the high points.

Re:Defense in Depth

[willdawg]

"...does anyone make a "locked-down by default" distro based off Red Hat/Debian/*BSD?"

How 'bout OpenBSD [openbsd.org]? Three years without a remote hole in the default install. Works for me.

Re:NAT != firewall

[Russ Nelson]

Who was talking about NAT? I'm suggesting that you run your public services on a public IP address and your private services on a non-routable private IP address.

The use of a firewall in itself offers little if any security!

Cool. Does it make me more correct if I use boldface?

And yes, geez, if you have one compromised host it can lead to other hosts being compromised. Should that surprise anyone?
-russ

What Story?

[torpor]

Are you referring to?

Re:Why a firewall?

[DirkGently]

Coupla reasons:

1. I'd say that they don't want to limit thier functionality. A tweaked firewall will let them keep useful schtuff turned on.

2.If the firewall uses its CPU to deflect the crap, then the web servers wont have to deal with it.

3. They have a BSD uberadmin who can make that BSD box walk the dog. If something else wierd goes on, it'll be in his back yard.

Dirk

Re:Defense in Depth

[DrgnDancer]

I read something a while back about a security team working for the Dept. of Defence (US) that designed and impleneted a set of scripts to "shoot back". The details are fuzzy, but basically this thing traced packets back to their source and "disabled" the attacking computer. The article I read did not contain many technical details, and I have since forgotten any it did have, but it was more concerned with legal issues.

In this case there was a legal problem because the thing had been implemented by the DOD (The US military is forbidden by law to take hostile action against US citizens except under specific circumstances), but here were plenty of other issues besides. It is way to easy to shoot "innocent" zombie computers for one thing. Just because a computer is attacking you doesn't mean the owner is, and counter attacking the computer is largely the same as attacking the owner (yes he SHOULD have secured his network, but when he sues you, will the courts accept that as a defence?). Even if you hit the right computer, you basically come out as guily as the attacker (there is no "self defence" clause in Information Security laws, and a "counter attack" is still an "attack"). He can insist that you be arrested at the same time he is, since you attacked him. There were some other things too, but like I said, it has been a while and the details are fuzzy.

Re:Defense in Depth

[Stinger]

I'd look at a distribution called Trustix [trustix.net]. This sounds like what your looking for.

Re:Why a firewall?

[ryanr]

Any local firewall (i.e. host protecting itself) will be inadequate if the source can be spoofed. I.e. ipchains can't tell the difference between a real NFS packet from 192.168.0.2, and a spoofed one. NFS runs over UDP in most cases, and the source would be easily spoofed. All it takes for your exploit to slip through would be to use the right source address, which would probably be easily determined.

You can do anti-spoofing somewhere external to the boxes, but theyn you've got an external firewall.

Re:Why are bogus packets getting to your routers?

[Fas Attarac]

In theory, this is correct.

However, in practice, we have incompetant admins, ignorant management, and underpowered hardware. In many backbone cases, ingress/egress filtering (or, indeed, most any kind of filtering) of these types of IP addresses isn't an option, due to the volume of data that these routers handle. They wouldn't be able to handle it. So, unfortunately, we must rely on ISP's on a more local scale to not only block these packets from coming in to the network (and likely on to their customers), but block them from leaving their network (or, perhaps, keep their customers from introducing them).

Along a similar line, these filters could/should be expanded to include the list of IP addresses that that network services. If done correctly (and down to an appropriate level of granularity), not only will all IP spoofing be eliminated, but anyone attempting to do so can be tracked down rather easily.

The fact that IP spoofing still shows absolutely no signs of abating is proof enough that few ISP's are filtering a damn thing.

Re:Why a firewall?

[Eivind Eklund]

Because Unix != Unix, as you should be well aware of. The Slashdot servers are AFAIK running Linux; what was installed in front of them was FreeBSD. FreeBSD has a number of features that make it better at handling attacks (dummynet, *_BANDLIM, etc) that I do not believe have equivalents for Linux. There are also some aspects of the basic kernel networking architecture in FreeBSD that might increase attack resistancy (but I suspect they do not make much difference for that case).

It might be possible to switch the main servers from Linux to FreeBSD, but as an interrim solution I think putting a FreeBSD firewall in front of them was a good tradeoff, giving time to evaluate whether an OS change on the servers themselves are warranted, or if there are reasons why keeping Linux make more sense.

Eivind.

Re:Curious about the setup...

[Kurt Gray]

For example, why are the servers serving images and static files segmented? Is there a lot you save from Apache configuration for dealing with one as compared to the other?

The web page servers run Apache+mod_perl+DBI+adsystem module, and the image servers run a much lighter Apache httpd with cache friendly headers.

Where does MySQL sit? Any "reason" behind Debian vs. RH other than "just because"?

MySQL is on the VA 3500 box which is also the Red Hat box. The servers all came with Red Hat and we installed Debian on them, expect the 3500, and I think that was because VA installed extra drivers and stuff we wanted to leave it as is.

Also, any chance you could go through some of the configuration choices made for your apache processes on each of these?

I think this will be in Rob's next post. If not we'll post in that forum.

Re:Why a firewall?

[PigleT]

"Why are you installing a Unix-based firewall in front of some Unix-based public servers? Why not secure the servers in the first place?"

It's called "security". The last thing you want is someone breaching your firewall and having instant access to your MySQL databases and everything. One (or more) sacrificial boxes facing forwards, critical stuff behind and an optional DMZ around the middle where you know exactly what's supposed to be going off.

It's also easier to administer these things if your logs are filterable on a per-hostname basis (if you want to do it that way) rather than having firewall things and local junk cluttering each other up on the same box.
~Tim
--
.|` Clouds cross the black moonlight,

Tracking DDOS or even DOS is difficult.

[Christopher Thomas]

That was a good account of what happened, but in part two, we want to hear what you are doing to track the bastards down.

Unfortunately, if I understand correctly, that can only be reliably done by manual traffic analysis by the sysadmins of the various routers en route, if I understand correctly. The origins and possibly routes of the incoming packets will have been forged, so you have to actually go from router to router looking for unusual traffic.

Disclaimer: I am not a networking guru.

Various modifications to routing software have been proposed that would make tracking easier (see the recent slashdot article [slashdot.org]). However, at present you're in for a lot of work and still probably out of luck.

Re:Why a firewall?

[orabidoo]

depends on your setup; if the server is its own firewall and is directly on the internet, then it has no reason to be getting private-IP packets at all. if it has two cards, one on the internet and one on a private IP, then you can do filtering based on the interface, which ipchains is perfectly able to do.

don't get me wrong, I see the point in having a separate firewall, which is to centralize security in one very secure machine with no services. but I don't think it's unreasonable to do it the other way either, if you only have a server or two to protect, and you have very tight control of their configurations.

Re:Why a firewall?

[YU Nicks NE Way]

First, any public system on the web should be behind a firewall. The amount of load that a firewall takes during an attack can easily drive even a very fast machine to 100% utilization; if you want your other servers to still be serving legitimate customers, you need a firewall.

Also, a firewall acts like a choke point -- any attack must pass through it. By monitoring the health of that one machine, you can monitor the health of the entire networks. In addition, if you want to allow remote administration of the items in the cluster, you can provide a secured path through the firewall; again, you have only the one point of failure.

It's usually wise to have stacked firewalls (an "airgap") in front of a popular site, though, and it's often best to use a variety of operating systems on those firewalls. Somehow, though, I can't see Slashdot doing the wise thing there, though, and putting a FreeBSD->W2K airgap at the front, with the Linux-based Slash behind it.

Re:Nice account, but who?

[Kenelson]

Actaully in the few times I have faced a DoS attacks, we did manage to track the users down. Just because they are forging the packets does not mean that their machine was able to avoid contacting the target completely.

In our case, we tracked the user down to his source by the "other" packets which he sent. The person sending the DoS often will send a ping and/or a name lookup of similar request prior to the attack or each time they add a new host in. Although it is a considerable exercise in collecting enough data to figure out which connects were real "valid" user contacts and which came from the kiddies. As a result we managed to isolate the DoS to specific hosts and subaddress ranges.

Of course, if you are into real fun assuming that you can get one of their target machines (which using a DoS scanner and a rough idea what subnet they are in) you can often port scan for eggdrop bots and other toys. Once you can convince a physical sysadmin to send you those files you then have a map of the kiddies entire bot and DoS network. Once I reach this stage I then post guards on IRC channels which their bots used and with a small ammount of detective work get their ISP. Script kiddies like to brag about what they do and it enevitably leads then them to surrender their identities.

In all the cases in which I managed to get that level of penitration into the kiddies network, I always managed to shut them down. ISP are very friendly about taking out malicious users especially when you supply logs and the attackers home address. :-) I even have gotten offers to have the attacker arrested (to bad I don't have the cash to fly there and file charges). Thus I can conclude although it is not an easly task it is not entirely necessarily impossible. (That is assuming your attackers are 14 year old kids and not paid professionals.)

--Karl

locked-down by default

[DreamerFi]

I'm not sure if this qualifies, but take a look at www.dubbele.com [dubbele.com]

Re:What's the Cisco angle?

[tzanger]

I disagree 100%. Knowledge of an installation's infrastructure should never comprimise the security of the setup. If it does, then you're relying (to a certain extent) on security through obscurity.

As I keep telling people around here, Security through obscurity is a valid security model WHEN ADDED TO OTHER SECURITY MEASURES!

What's wrong about adding a little obscurity to your design? It makes it More secure than before. Not much, admittedly, but every layer counts and if your traffic logger is picking up a guy scanning around trying to get past your thin obscurity layer, it can set off alarms earlier than without.

I AM NOT saying that security through obscurity is a valid measure when used alone, but when used in conjunction with other security methods, it enhances overall security.

Do you post a sign on your house saying "My locks are Master model ES014, 6-pin with 1/8" slot width?" No, you don't put anything. Hence another layer of security (obscurity) on top of the physical lock. Similarly if you did have such a sign and a would-be thief knew about a trick for that exact model lock, you've just made his day easier. Why tell the world that you're using BSD/Linux/NT version x with software a, b and c? You're just eliminating options that they would-be attacker would otherwise have to try!

Security should be provided by a well thought out layered approach: network layering (multiple firewalls, screening routers, IDS, etc...), host-based security (tcp wrappers, service minimalization & replacement, tripwire, etc..), and application security (ie. authentication, verification, etc...)

Yes, and then don't go about advertising the exact methods you've used to lock yourself down. Keep 'em guessing long enough to trip something and alert you, and make sure the security model is thick enough to keep him at bay.

10.0.0.0 net

[Signal 11]

I've found this alarmingly common to be routed on networks. I wish router manufacturers would squish this once and for all - it's in the RFCs that these are NOT to be routed on the 'net at large.

I've had alot of portscans for 31337 and 12345 in the past week on the mediaone network, all from 10.0.0.0/16 networks. I am massively annoyed that they let this through and block ports 137:139. Umm.. is this solving the problem? No! Oh, and they've taken a liking to scanning their customers boxen.. but I digress.

DDoS is the direct result of sloppy upstream administrators. IF I were in your shoes, I would be suing every person upstream for atleast a few hops for passing those 10.0.0.0 packets along for gross negligence.

Re:What's the Cisco angle?

[flibbertigibbet]

Why the heck would they need that? They don't need stateful inspection of every packet or a bunch of application proxies.

They can instead install FreeBSD, rate limit SYN and ICMP (echo, echo-rep, or everything), block all SYN and ICMP right out from reserved and invalid IP ranges, and then drop in rules using IPfilter or ipfw when needed. You can do hundreds of mb/s on this setup.

I have this and more, including automa set up to establish damaging patterns, using my logging mechanism (no, I'm not writing to a file and creating a DoS for myself).

Part2: Can we get some pictures?

[a.out]

I know I'm not the only one who would like to see pictures of this whole setup :)

And while you're at it get CowboyNeal to give us a sexy pose *on* the servers (grin)

Re:Why not a firewall.

[Russ Nelson]

Machines are cheaper than people. It's easier to configure N+1 machines all the same than to configure N machines one way and one machine a different way.

Not ignoring, just forgetting to dispense with it as an issue.
-russ

Re:Why not a firewall.

[Russ Nelson]

It's not that I'm a god. It's just that I've seen firewalls and the machines behind them, and I'm unimpressed by the way they work, and I'm unimpressed by the arguments for them.

Why isn't your router blocking traffic with an unroutable source address?

You mean they build insecure boxes and then put them on the net? Why did they waste their time?
-russ

How does a DDoS attack screw up the / code???

[moopster]

Every time I tried to view the front page of /. it came up waaaaaaay funky. 1 - Did anyone else experiance this? 2 - Is there more to this problem then just a DDoS? mcd

What about the children?

[Anonymous Coward]

While I agree that the Slashdot DDoS attack caused many people quite a bit of annoyance and frustration, I think leaving the impact at that is very short sighted.

Firstly, I don't think the blame for this DDoS can be centered on just one person or group. Obviously, those who attacked Slashdot are to blame, as are Slashdot's sysadmins, and the people at Arrowpoint. And secondly, the costs of this are much greater than you might think.

I have an eight year old daughter. We had a family pet - a rabbit, black, named Midnight, and my daughter was very fond of it. Midnight, sadly, passed away about two months ago. A week or two after Midnight died, my daughter came to me in tears and asked me, "Daddy, why won't God bring Midnight back? I've been praying like Deacon Simmons told me to."

Naturally, I had to think about how to respond to this. I finally answered, "well, honey, God is a little like Slashdot. He can seem arbitrary, cruel, and unresponsive, but he's really a nice guy who's just a little out of touch and is a little slow at responding to requessts."

This was fine, and I thought that would be the end of it. However, when Slashdot went down last week, my daughter burst into my den, positively sobbing and wailing, and managed to choke out "Daddy! Daddy! I can't get to Slashdot!" "Honey," I said, "it's just a website." But, between sobs, she said, "but you said God is just like Slashdot, remember? Does this mean God is dead?"

I tried to console her as best I could, but nothing seemed to work. When Slashdot came back up, she seemed to return to normal, but she hasn't been quite the same since. She doesn't ask me about God so much any more, and she seems less interested in Church.

As a good Christian, I will turn the other cheek, and not call for the punishment of those responsible. But to the heinous criminals and negligents responsible for this, I must ask, how do you feel about destroying a small girl's sense of innocence and wonder about the world? About crushing her childish dreams and idealism? About shattering her faith in God and his benevolence? About possibly having crushed her soul and emotion forever, leaving her to live the rest of her days in spiritual agony as a broken, scarred husk of a person?

I hope all of you think long and hard about what you've done. What is the soul of a child worth, next to a few double-checks of the router?

Thank you.

No doubt it was MS/MPAA/RIAA/Metallica/Dr Dre

[auntfloyd]

I'm sure that these great enemies of the Slashdot Empire have found this to be a convenient time to strike. We must systematically seek and destroy all those suspected of having sympathies with the MPAA, RIAA, or Microsoft for security reasons.

Therefore, all

Windows users

CD listeners

Movie watchers

Metallica fans

are asked to please leave now or face prosecution.

thank you.

Re:NAT != firewall

[Russ Nelson]

Which operating systems forward source routed packets or tunnel packets without explicitly being configured to do that?

You say it's weak security, but you come up with a weak example of why it is.
-russ

Re:What Story?

[Rendus]

We're talking about the infamous Sengan disabling comments on the US bombing Iraq story. That's what this part of the thread is about anyway.

If you were refering to my TCWWW statement, at one point Rob (Malda) had DNS screwed and we couldn't get to slashdot if you put the www. in front of it. He left it like that for quite some time, and he'd post articles that link to /. and say something like "But they used The Cursed WWW", and hence TCWWW.

Re:Of course it was the router!

[Chang]

MSFC stands for "Multilayer switch feature card". This is the part of a Cisco switch/router that makes it do layer-3 switching (routing at wire speeds).

Re:Why not a firewall.

[Russ Nelson]

No, I'm not trolling. I haven't seen an rationale for a firewall which is any better than "Well, we're too stupid and lazy to lock down N Unix hosts, so we're going to lock down one. Somehow we will become less stupid and lazy because there is only one machine."

If I can secure a firewall that I control, then I can secure a firewall that I control.

If X then X is true every time, but it's not much of an argument for a firewall.

I can't prevent the group behind the firewall from introducing vulnerabilities on their side of the street

If they're in public services, you're toast *anyway*, because your firewall is letting those services through. If they're in private services, then why for God's sake did you bind them to a public IP address???

Most of the things that people are using firewalls to protect against can be solved by using non-routable IP addresses and some small amount of filtering on your router.
-russ

"Shoot Back" is recipe for disaster - builtin DDOS

[billstewart]

It's one thing to let experts shoot back. It's another thing to make it a widely distributed capability, especially an automated one. Currently a bad guy who wants to run a DDOS needs to crack a few hundred poorly-run machines and then fire up his scripts to abuse them. But if "shoot-back" tools are widely distributed, all he needs to do is find how to forge an attack in a way that will convince a particular shootback tool to attack some victim, and then spam out as many attacks as necessary to get the shootbacks to overwhelm the victim. (Obviously it's still worth doing this from a cracked machine, but you don't have to own a lot of cracked machines to obfuscate yourself.)

This is different from mostly-passive traps like teergrube (FAQ [iks-jena.de]; jargon [tuxedo.org]) or Deception Toolkit [all.net] or spider traps which sit around waiting for Bad Guys to attack them and react unexpectedly when attacked (e.g. ...res.p...o...n...d....v...e...r...y....s..l..o.. o...ooo...w...l...y.... while logging stuff or sending back odd replies). ("mostly passive" doesn't exclude leaving lots of inviting copies of your address around for harvesters or script kiddies to find.)

Re:Owned? - Nope

[Cpyder]

Maybe you should type more carefully, since you
requested http://slahsdot.org (slaHSdot) not
slashdot.org...

I registered that domain (for free @ namezero) to
help the people who couldn't type. Sorry if I scared you :-)

[e-mail]
_
/ /pyder.....
\_\ sig under construction

Is that what took so long..

[ch-chuck]

waiting for Rob to toggle in a boot loader to IPL from the punch card reader?

RSM/MSFC definations

[Ed Bugg]

RSM - Route Switch Module
- Basically a router on a card in the switch for routing between VLANs

MSFC - Multilayer Switch Feature Card
- Once a route for a packet flow is figured out (from the first packet going through the router) all other packets from the flow get switched instead of routed.

Re:how is this a troll???

[Russ Nelson]

And FreeBSD is immune to this effect? How can this be? Even if 50% of all FreeBSD users are experts, and 10% of all Linux users are experts, there is still (as I said earlier and it's still not a troll) more Linux expertise.
-russ

Re:What's the Cisco angle?[ERRATTA]

[John Fulmer]

Sorry. Make that mid 90's...

jf

A little more detail on the hardware setup

[Kurt Gray]

Rob is going to post exact hardware specs later, but in the meantime just to give you a brief idea where the "Arrowpoint" sits in relation to all this... Slashdot now running on several machines, all VA FullOns, running Debian and few running Red Hat, Apache+mod_perl, MySQL. The database is on its own VA 3500 server. There are currently six VA FullOns serving web pages from an NFS server, and three other web servers serving images.

All of these machines were behind an Arrowpoint (CS-100) firewall/load balancer which took it on the chin when we got DDoSed, so basically the Arrowpoint was taking the full force of the attack. So as described above we replaced it with a CS-800 and a BSD firewall.

I guess we learned that if you're going to post a letter from a Microsoft attorney on your web site the same day you implement a few new troll filters you better be prepared for the fury of hell to rain down on you. Then again this is Slashdot, so we always should be prepared for the fury of hell to rain down on us.

DDOS != 10.0.0.0

[Christopher Thomas]

DDoS is the direct result of sloppy upstream administrators. IF I were in your shoes, I would be suing every person upstream for atleast a few hops for passing those 10.0.0.0 packets along for gross negligence.

Um, no.

DDOS simply requires that a lot of compromized boxes be able to send you packets. Spoofing to non-existant return addresses is an orthogonal issue. You reply that it's used to mask the souce boxes? Any _valid_ address could also be used for that, so filtering would gain you nothing against that.

I agree that filtering of reserved addresses should be done, but that would not hinder a DDOS attack.

A link or two

[Praxxus]

Charles Spurgeon's Ethernet Web Site [utexas.edu]

Jason Schwarz Ethernet Tutorial [lothlorien.net]

Lantronix Networking Tutorials [lantronix.com]

You might also try typing "ethernet tutorial" or somesuch in your favorite web search engine. Hope this helps!

--

Re:Could someone point me to a networking tutorial

[Fishstick]

I would recommend you start here:
net3-4-howto [tucows.com]
firewall howto [tucows.com]
masq-howto [tucows.com]

I have also heard that you can directly connect two NICs with a special cable. Do you need software changes to do this?

Yes, you can do this with a crossover cable and no you don't really need any special software to do this. I use one when I bring my laptop into work and want to hook it to my workstation. You can either make one yourself or buy one at any decent site like hardwarestreet.com.

Sorry I am so clueless.

:-) Try 'Networking for Dummies'. It is a pretty good reference for setting up a Q&D network. The examples are for windows, but the basic principles are the same. I started out with the intent to hook up my PC with my Wife's to share a printer, knew nothing at all about setting up a LAN. That book and those howto's and a lot of tinkering were pretty much all that were required. Now I have my whole house wired, I have a Linux box hooked up to my cable modem doing masquerading for the machines in my house. I set up a server to do SMB file and print sharing and stuff.

Anyway, good luck.

Re:What's the Cisco angle?

[John Fulmer]

I'm not a slashdot admin (but I could play one on TV!), but I am painfully familiar with PIXs.

The idea behind the PIX, or any firewall-like object, is to allow 'good' traffic (http, smtp, etc) into the production network, and reject 'bad' traffic (oddball ports, like port 0, unauthorized UDP traffic, etc).

The problem with the PIX, is that it is essentially a fairly stupid router that can do network address translation and other bells and whistles, but it does it poorly. VERY poorly. It was designed as a network address translation system back in the mid 80's (anyone remember all the "We'll run out of IP's by 1997!") by a company that Cisco later bought. Cisco took the product, did a logic problem ( "Firewalls can do address translation. PIX does address translation. PIX is a firewall!"), and had themselves a firewall.

Its configuration makes a lot of sense to someone familiar with cisco router ACL rules, but no one else.

They are probably much better off with the BSD box. Although it's not a good idea to advertise their security infrastructure layout to the world. (Hint, Hint, CmdrTaco!)

jf

Re:Why a firewall?

[YU Nicks NE Way]

First, in the event of an attack, a single point of failure isn't necessarily a bad thing. If you know exactly what has fallen over, you're more than half way to knowing how to fix it. A firewall is easy to secure precisely because it isn't a general purpose box; the BOFH knows exactly what's running on it. The worker bees behind the firewall are a different matter; they presumably run a wide variety of different software. Failure analysis becomes much more complicated. (Not to mention that diverse software allows for interaction among the different components, which exposes flaws.)

In a DDOS attack, if your firewall falls over, then the odds are that your network would have fallen over, too. Slashdot "only" handles 100Mb/sec, though -- one high end machine should be able to handle a pipe that wide. But, if the pipe gets wider, then they can get a virtual "choke" with a load balancer in front of the firewall.

Re:PIX inusfficient?

[gavinhall]

Posted by BSD-Pat:

The problem here is that we only had one subnet to work with. The PIX we had wouldn;t to the type of filtering/bridging that I wanted.

Cisco wants a DMZ on these things.

I needed a bridge...why I didn't use linux...

It was quicker and easier for me... ipchains has always been a pain in my arse... ipfw and ipfilter I know best.

The other thig is that we fried an arrowpoint cs-100 (little itty bitty dinky thing that was being replaced with a bigger one)

the little arrowpoint couldn't take the traffic of 109Mbits , it wasn;t meant for that, we were waiting on arrowpoint to ship us the unit we were *supposed* to have.

*BSD fills the gap because I know it inside and out, and it was the quickest to get up at that point.

As far as the router, we can't do any type of stateful filtering on the 6509, due to some setup that exodus has with the HSRP stuff, I'm sure given enough thought I could figure out how to do it, however we were running on crisis mode.

The BSD firewall filled that gap for us...I can now do access lists on that, instead of the cisco.

and we still have a "DMZ" but its on the same subnet.

The arrowpoint CS-800 was emergency shipped to us that afternoon....its about as big as a cisco 6509...and ummm won't die under that type of traffic/content checking (its layer 5 remember)

-Pat

Prevent SYN floods @ the MSFC

[Anonymous Coward]

Why not do IOS Load Balancing from the 6500/MSFC itself? You can use SYNGuard with the Load Balancing to protect against SYN floods... refer to: http://www.cisco.com/univercd/cc/td/doc/product/so ftware/ios121/121newft/121limit/121e/121 e1/iosslb1.htm#xtocid446613 -- Anonymous Cisco Employee

Re:Why a firewall?

[Thomas Charron]

Most of the time, you'll have a backup firewall just waiting to be turned on. Load is only *part* of the issue that occurs. The first one, which is simple enough, is that it's easier to secure a box on the front end then one on the backend. My home firewall, and the one at work, quite literally have *NO* open ports at all. We really hope the console never dies, becouse we're forked if it does.. :-P But doing this was *really* easy. Now, a backend box is only as secure as it's weakest app, and in the case of things such as MySQL, Samba, etc, it simply makes more sense to have at least one box sitting in front with *everything* locked down. 'Spec when individuals use a system such as Linux's ability to IpMASQ. A good situation would be a front end firewall doing IP forwarding to a load balancing system. Have it only forward the ports required, aka, port 80. Anything else, rejected and logged. Then, even if they do manage to exploit something in your web server, they magically find themselves unable to telnet to that trojan backdoor they managed to get running.

The more hurdles you put in front of the kidies, the more likely they are to get bored..

Re:Anti-spoof filters on the Exodus network

[gavinhall]

Posted by BSD-Pat:

even more wacky, we were getting stuff from 0.0.0.0/8 (gee, how the F#@% do you filter that??!?!) lets filter the equivalent of "any", gee...

we have been talking to Exodus to get this problem resolved.

Re:DDOS != 10.0.0.0

[InsaneGeek]

I believe that he was more going along the line of things outgoing packets that have a source address from outside my network should be dropped before it gets outside your own network (not just the reserved, but anything that isn't supposed to be outgoing over that router).

As long as you aren't in wild and wooly peering arangements, one should be able to know all the ipaddress that are inside ones network (and within each segment of the network). Once a router sees something that can't possibly be coming from inside that network, it should be dropped and throw up alarms, bells, flashing lights, etc. cause something just ain't right (either a misconfigured client or someone trying something bad).

Doing this type of filtering doesn't prevent your system from being used in a DDOS attack, but it prevents your system from being used in the attack with a spoofed address. Hence see 50mb/sec from host w.x.y.z, contact owner of that address block and get it stopped, since it is not forged they have a compromised box internally. If everybody started doing that the world would be a MUCH better place to live in.