Introducing The New Slashdot Setup

At the request of countless users, we're happy to finally present a summary of the new setup over at Exodus. It's the result of over 6 months of work from a lot of people, so shadouts to Adam, Kurt, and Scoop, Team P:Pudge, PatG & Pater for the code, and Martin BSD-Pat and Liz for getting the hardware and co-loc taken care of. Now hit the link below and see what these guys did:

the original version of this document was written by Andover.Net Alpha Geek Kurt Grey. The funny jokes are his. The stupid jokes are mine.

The Backstory
We realized soon that our setup at Digital Nation was very flawed. We were having great difficulty administering the machines and making changes. But the real problem was that all the SQL traffic was flowing over the same switch. The decision was made to move to Exodus to solve these problems, as well as to go to a provider that would allow us to scatter multiple data centers around the world when we were ready to do so.

Meanwhile Slashcode kicked and screamed its way to v1.0 at the iron fists of Pudge (Chris Nandor) and CaptTofu (Patrick Galbraith). The list of bugfixes stretches many miles, and the world rejoiced, although Slashdot itself continued to run the old code until we made the move.

The Co-Loc
Slashdot's new co-location site is now at Andover.Net's own (pinky finger to the mouth) $1 million dedicated datacenter at the Exodus network facility in Waltham, Mass, which has the added advantage of being less than 30 minute drive for most of our network admins -- so they don't have to fly cross-country to install machines. We have some racks sitting at Exodus. All boxes are networked together through a Cisco 6509 w/ 2 MSFCs and a Cisco 3500 so we can rearrange our internal network topology just by reconfiguring the switch. Internet connectivity to/from the outside world all flows through an Arrowpoint CS-800 (which replaced the CS-100 that blew up last week) switch which acts as both a firewall load balancer for the front end Web servers. It also so happens that the Arrowpoint shares the same office building with Andover.Net in Acton so whenever we need Arrowpoint tech support we just walk upstairs and talk to the engineers. Like, say, last week when the 100 blew up ;)

The Hardware

• 5 load balanced Web servers dedicated to pages
• 3 load balanced Web servers dedicated to images
• 1 SQL server
• 1 NFS Server

All the boxes are VA Linux Systems FullOns running Debian (except for the SQL box). Each box (except for the SQL box) has LVD SCSI w/ 10,000 RPM drives. And they all have 2 Intel EtherExpress 100 LAN adapters.

The Software
Slashdot itself is finally running the latest release of Slashcode (it was pretty amusing being out of date with our own code: for nearly a year the code release lagged behind Slashdot, but my how the tables have turned).

Slashcode itself is based on Apache, mod_perl and MySQL. The MySQL and Apache configs are still being tweaked -- part of the trick is to keep the MaxClients setting in httpd.conf on each web server low enough to not overwhelm the connection limits of database, which in turn depends on the process limits of the kernel, which can all be tweaked until a state of perfect zen balance has been achieved ... this is one of the trickier parts. Run 'ab' (the apache bench tool) with a few different settings, then tweak SQL a bit. Repeat. Tweak httpd a bit. Repeat. Drink coffee. Repeat until dead. And every time you add or change hardware, you start over!

The Adfu ad system has been replaced with a small Apache module written in C for better performance, and that too will be open sourced When It's Ready (tm). This was done to make things consistant across all of Andover.Net (I personally prefer Adfu, but since I'm not the one who has to read the reports and maintain the list of ads, I don't really care what Slashdot runs).

Fault tolerance was a big issue. We've started by load balancing anything that could easily be balanced, but balancing MySQL is harder. We're funding development efforts with the MySQL team to add database replication and rollback capabilities to MySQL (these improvements will of course be rolled into the normal MySQL release as well).

We're also developing some in-house software (code named "Oddessey") that will keep each Slashdot box sychronized with a hot-spare box, so in case a box suddenly dies it will automatically be replaced with a hot-spare box -- kind of a RAID-for-servers solution (imagine... a Beuwolf cluster of these? *rimshot*) Yes, when it'll also be released as open source when its functional.

Security Measures
The Matrix sits behind a firewalling BSD box and an Arrowpoint Load balancer. Each filters certain kinds of attacks and frees up the httpd boxes to concentrate on just serving httpd and allows the dedicated hardware to do what it does best. All administrative access is made through a VPN (which is just another box).

Hardware Details

Type I (web server)
VA Full On 2x2 Debian Linux frozen
PIII/600 Mhz 512K cache
1 GB RAM
9.1GB LVD SCSI w/ hot swap backplane
Intel EtherExpress Pro (built-in on moboard)
Intel EtherExpress 100 adapter

Type II (kernel NFS w/ kernel locking)
VA Full On 2x2
Debian Linux frozen
Dual PIII/600 Mhz
2 GB RAM
(2) 9.1GB LVD SCSI w/ hot swap backplane
Intel EtherExpress Pro (built-in on moboard)
Intel EtherExpress 100 adapter

Type III (SQL)
VA Research 3500
Red Hat Linux 6.2 (final release + tweaks)
Quad Xeon 550 Mhz, 1MB cache
2 GB RAM
6 LVD disks, 10000 RPM (1 system disk, 5 disks for RAID5)
Mylex Extreme RAID controller 16 MB cache
Intel EtherExpress Pro (built-in on moboard)
Intel EtherExpress 100 adapter

Re:The Return of the Server

while (<NICEPOST>) {
s/KURT/MARTIN/;
s/Andover/Adam/;
}

ROBLIMO: Not after we demonstrate the power of this station. In a way,
you have determined the choice of the web site that'll be slashdotted
first. Since you are reluctant to provide us with a URL, I have chosen
to test this station's slashdotting power...
on your home page on iVillage!

AC: No! iVillage is peaceful. We don't flame Linux on iVillage.
We only discuss travel and mystery novels. You can't possibly...

ROBLIMO: You would prefer another target? A commercial target? Then name the URL!

Roblimo waves menacingly toward AC.

ROBLIMO: I grow tired of asking this. So it'll be the last time. What is the URL?

AC: (softly) pcweek.com.

AC lowers her head.

AC: The FUD piece was posted on pcweek.com.

ROBLIMO: There. You see Lord Taco, she can be reasonable. (addressing
Hemos) Continue with the operation. You may post when ready.

The Return of the Server

INT EXODUS - MAIN SERVER ROOM

[MOFF KURT, a tall, confident technocrat, strides through the assembled geeks to the base of the shuttle ramp. The geeks snap to attention; many are uneasy about the new arrival. But Moff Kurt stands arrogantly tall.]

[The exit hatch of the shuttle opens with a WHOOSH, revealing only darkness. Then, heavy FOOTSTEPS AND MECHANICAL BREATHING. From this black void appears DARTH TACO, LORD OF THE SITH. Taco looks over the assemblage as he walks down the ramp.]

MOFF KURT:
"Lord Taco, this is an unexpected pleasure.
We're honored by your presence."

DARTH TACO:
"You may dispense with the pleasantries, Commander. I'm here to put you back on schedule."

[The commander turns ashen and begins to shake.]

MOFF KURT:
"I assure you, Lord Taco, my men are working as fast as they can."

DARTH TACO:
"Perhaps I can find new ways to motivate them."

MOFF KURT:
"I assure you, this station will be operational
as planned."

DARTH TACO:
"Andover does not share your optimistic appraisal of the situation."

MOFF KURT:
"But he asks the impossible. I need more geeks."

DARTH TACO:
"Then perhaps you can tell them when they arrive."

MOFF KURT: [aghast]
Andover's coming here?

DARTH TACO:
"That is correct, Commander. And they are most displeased with your apparent lack of progress."

MOFF KURT:
"We shall double our efforts."

DARTH TACO:
"I hope so, Commander, for your sake. Andover is not as forgiving as I am."

"This server is now the ultimate power in the universe. I suggest we use it!"

Re:Why spend all that $ to fix MySQL?

In MySQL you can decide per table if you want the table to be fast or take the speed penalty of making it transaction safe.

In MySQL, you do not have the choice turning on transactions and atomicity.

You have the choice of turning on features that they mistakenly label transactions and atomicity, but let's call a spade a spade here.

You use MySQL if you care about speed a lot, and don't care much about data integrity. That's a perfectly valid position, but let's not pretend it's some other position.

If you do care about data integrity, you use something other than MySQL, and find another way to achieve the speed.

--

Beware the Intel EtherExpress Pro w/linux

We have several Fullons ourselves, and under heavy load these cards cease to function. I have to log in via an internal network interface and ifconfig the outside IP down then back up.

Slashdot Commerical? :)

Webservers: $22525

NFS servers: $21120

Database server: $25739

Being THE place for Natalie Portman and Hot Grits on the Web: priceless

There's some things money can't buy. For everything else, there's Slashdot.

---

Re:Why spend all that $ to fix MySQL?

Be careful of MySQL documentation, and web site ... over the past couple of years, the PostgreSQL developers have tried to work with the MySQL folks to improve their various pages, especially their 'comparison' page ... we recently sent them a list of changes (the number of them eludes me, but it was alot) on just one section of the comparison page that listed a bunch of their types as 'SQL-Compliant' that have nothing to do with the standard, and a bunch of 'SQL Non-Compliant' that are required by the Standard ... As for 'a magnitude speed difference' ... check out PostgreSQL now. I would imagine that their 'tests' were based on several releases behind ... v7.0, that was just released, is several times faster then our last release, which was noticeably faster then v6.4 ... The point I'm making is that if you are going to judge something, judge it on your experiences, not on a competing vendors experiences... you might just be pleasantly surprised ... - [e-mail]

Re:Important Question...

Well, a typical Exodus facility isn't nuke-proof, but it's pretty damn close. I've toured one (in Herndon, VA) because our company is about to co-loc at it. Here's a brief rundown of the physical security:

• The building is a converted warehouse; every interior and exterior wall is filled with reinforced concrete
• Extra curbs and islands in parking lot directly in front of the door
• Two-foot-high concrete pillars in sidewalk leading to the door
• Concrete overhang over door, only 8 feet of clearance
• No windows
• One door for entry (no loading dock)
• Receptionists/security chimps sit in a bulletproof enclosure
• Mantrap-style "airlock" with 3"-thick steel doors leads to all computer facilities; office areas have single steel doors sealing them off from lobby

You run into all this before you even see anything resembling a computer, apart from the terminals in the receptionist's enclosure. In the actual computer pens, you have the cages, and for the really paranoid, you can get a steel box with a biometric lock instead of a conventional cage.

To sum up...it would take a truly concerted effort to physically breach one of these facilities.

Aero

Why Debian/RH?

Forgive me if this has been asked elsewhere, but why did y'all choose those distributions for those servers? I'm genuinely curious; I'm unfamiliar with the large-scale differences between distributions. (My computer runs Mandrake... that decision was based on the single factor that my friend happened to have a Mandrake CD on him.)

Why spend all that $ to fix MySQL?

Fault tolerance was a big issue. We've started by load balancing anything that could easily be balanced, but balancing MySQL is harder. We're funding development efforts with the MySQL team to add database replication and rollback capabilities to MySQL (these improvements will of course be rolled into the normal MySQL release as well).

Just out of curiosity, wouldn't it be easier to use something like PostgreSQL [postgresql.org] (which is just as freely available) that already has rollback & atomicity than to pay the MySQL people to develop it? Didn't y'all read the article on here a few weeks ago, "Why not MySQL? [openacs.org]"

__________________________________________________ ___

Re:For the record -- "Exodus"??

I'd tell you, but I have a personal policy against helping lazy luddites who think they're taking some kind of principled stand because they don't visit sites that use <img> tags. At least read the damn FAQ linked to on their home page.

Cheers,
[e-mail]

Re:MySQL Server.

This isn't flamebait - I'm honestly wondering why you wouldn't go away from MySQL here. It is obvious that money isn't a concern - so why not fork over the money for some Oracle licenses and a competent Oracle admin?

It's not like it is for some sort of open-source reason - MySQL isn't released under an open-source license. I'm curious why slashdot/Andover are spending money funding a closed source project rather than funding an open-source one or forking over the $ for a more capable database like Oracle.

Exodus is a *BIG* ISP

Who/what exactly is Exodus?

Exodus is one of the world's biggest (in terms of service capacity available) Internet Service Providers.

"We're going to need bandwidth. Lots of bandwidth."

Exodus specializes in having more bandwidth then most of the third world. They've got NAPs (Network Access Points, i.e., backbone connections) all over the continental United States, and a few outside the US as well. They link this all together using both external and internal networks. The end result is, most anywhere on the net that has a good connection, has a good connection to Exodus.

They provide servers. Do you need to host downloads for ten million users? Exodus can give you servers to do so.

They provide co-location space. If their standard server packages just won't cut it -- bring your own. They'll give you a rack, a dedicated co-loc cage, or a dedicated high security vault.

Their web page [exodus.net] has a lot of graphics because they have a lot of pictures of their equipment and graphs of their capacity. It is actually justified. You may want to make a return trip.

Debian not on the SQL server?

There has been a small talk about this story in the debian-devel channel on IRC just some minutes ago, and of course the great question was:
Why isn't the SQL server Debian as well?
If there's any problem with Potato's MySQL, I think Debian would be pleased to hear, whether it's a bug report in the BTS or whatever.

Thanks

Re:For the record -- "Exodus"??

Others have said it well, but I'll add this: Exodus hosts Yahoo. 'Nuff said.

--

Lies

You're using Win2000, IIS, and Active Server Pages. We all know it. Quit making stuff up.

Network Topology

I don't know about the other slashdottes, but I for one would love to see how the Slashdot network is configured topologically.