Slashdot Log In
Handling the Loads
from the when-it-all-hits-the-fan dept.
I woke up and it seemed like a normal day. Around 8:30 I got to the office and made a pot of coffee. I hopped on IRC, started rummaging through the submissions bin, and of course, began reading my mail. Within minutes someone told me on IRC what had happened just moments after the impact of the first plane. Just a minute or 2 later, submissions started streaming into the bin. And at 9:12 a.m. Eastern Time, I made the decision to cancel Slashdot's normal daily coverage of "News for Nerds, Stuff that Matters," and instead focus on something more important then anything we had ever covered.
I couldn't get to CNN, and MSBNC loaded only enough to show me my first picture of the tragedy. I posted whatever facts we had: these were coming from random links over the net, and from Howard Stern who syndicates live from NY, even to my town. Over the next hour I updated the story as events happened. I updated when the towers collapsed. And the number of comments exploded as readers expressed their outrage, sadness, and confusion following the tragedy.
Not surprisingly, the load on Slashdot began to swell dramatically. Normally at 9:30 a.m., Slashdot is serving 18-20 pages a second. By 10 we were up to 30 and spiking to 40. This is when we started having problems.
At this point Jamie and Pudge were online and we started trying to sort out what we could do. The database crashed and Jamie went into action bringing it back up. I called Krow: he's on Western time, but he knows the DB best, and I had to wake him up. But worst of all, I had to tell him what had happened in New York. It was one of the strangest things I've ever done: it still hadn't settled in. I had seen a few grainy photos but I don't have a TV in my office and hadn't yet seen any of the footage. After I hung up the phone I almost broke down. It was the first time, but not the last.
The DB problem was a known bug and the decision was made to switch to the backup box. This machine was a replicated mirror of Slashdot, but running a newer version of MySQL. We hadn't switched the live box simply because it meant taking the site down for a few minutes. Well we were down anyway, and the box was a complete replica of the live DB, so we quickly moved.
At this point the DB stopped being a bottleneck, and we started to notice new rate limits on the performance of the 6 web servers themselves. Recently we fixed a glitch with Apache::SizeLimit: Functionally, it kills httpd processes that use more then a certain amount of memory, but the size limit was to low and processes were dying after serving just a few requests. This was complicated by the fact that the first story quickly swelled to more than a thousand comments ... we've tuned our caching to Slashdot's normal traffic: 5000-6000 comments a day, with stories having 200-500 comments. And this was definitely not the normal story. Our cache simply wasn't ready to handle this.
Our httpd processes cache a lot of data: this reduces hits to the database and just generally makes everything better. We turned down the number of httpd processes (From 60 on each machine, to 40) and increased the RAM that each process could use up (From 30 to 40 and later 45 megs) We also turned off reverse hostname lookups which we use for geotargetting ads: The time required to do the rdns is fine under normal load, but under huge loads we need that extra second to keep up with the primary job: spitting out pages as fast as possible.
This was around noon or so. I was keeping a close eye on the DB and we noticed a few queries that were taking a little too long. Jamie went in and switched our search from our own internal search, to hitting Google: Search is a somewhat expensive call on our end right now, and this was necessary just to make sure that we could keep up. We were serving 40-50 pages/second ... twice our usual peak loads of around "Just" 25 pages a second. I drove the 10 minutes to get home so I could watch CNN and keep up better with what was happening.
We trimmed a few minor functions out temporarily just to reduce the number of updates going to frequently read tables. But it was just not enough: The database was now beginning to be overworked and page views were slowing down. The homepage was full of discussions that were 3-4x the average size. The solution was to drop a few boxes from generating dynamic pages to serving static ones.
Let me explain: most people (around 60-70%) view the same content. They read the homepage and the 15 or so stories on the homepage. And they never mess with thresholds and filters and logins. In fact, when we have technical problems, we serve static pages. They don't require any database load, and the apache processes use very little memory. So for the next few hours, we ran with 4 of our boxes serving dynamic pages, and 2 serving static. This meant that 60-70% of people would never notice, and the others would only be affected when they tried to save something ... and then they would only notice if they hit a static box, which would happen only one in 3 times. It's not the ideal solution, but at this point we were serving 60-70 pages a second: 3x our usual traffic, and twice what we designed the system for. We got a lot of good data and found a lot of bottlenecks, so next time something that causes our traffic to triple, we'll be much more prepared.
At the end of the day we had served nearly 3 million pages -- almost twice our previous record of 1.6M, and far more then our daily average of 1.4M. During the peak hours, average page serving time slowed by just 2 seconds per page ... and over 8000 comments were posted in about 12 hours, and 15,000 in 48 hours.
On Wed. we started to put additional web servers into the pool, but that ended up not being necessary. We stayed dynamic and had no real problems on all 6 boxes all day. We peaked at around 35-40 pages/second. We served about 2 million pages. Thursday traffic loads were high, but relatively normal.
Summary So here is what we learned from the experience.
- We have great readers. I had only one single flame emailed to me in 24 hours, and countless notes of thanks and appreciation. We were all frazzled over here and your words of encouragement meant so much. You'll never know.
- Slashteam kicks butt. Jamie, Pudge, Krow, Yazz, Cliff, Michael, Jamie, Timothy, CowboyNeal, you guys all rocked. From collecting links to monitoring servers, to fixing bits of code in real time. It was good seeing the team function together so well ... I can't begin to describe the strangess of seeing 2 seperate discussions in our channel: one about keeping servers working, and another about bombs, terrorists, and war. But through it all these guys each did their part.
- Slash is getting really excellent. With tweaks that we learned from this, I think that our setup will soon be able to handle a quarter million pages an hour. In other words, it should handle 3x Slashdot's usual load, without any additional hardware. And with a more monstrous database, who knows how far it could scale.
- Watch out for Apache::SizeLimit if you are doing Caching.
- Writing and reading to the same innodb MySQL tables can be done since it does row-level locking. But as load increases, it can start being less then desirable.
- A layer of proxy is desirable so we could send static requests to a box tuned for static pages. For a long time now we've known that this was important, but its a tricky task. But it is super necessary for us to increase the size of caches in order to ease DB load and speed up page generation time ... but along with that we need to make sure that pages that don't use those caches don't hog precious apache forks that have them. Currently only images are served seperately, but anonymous homepages, xml, rdf, and many other pages could easily be handled by a stripped down process.
What happened on Tuesday was a terrible tragedy. I'm not a very emotional person but I still keep getting choked up when I see some new heart breaking photo, or a new camera angle, learn some new bit of heart breaking information, or read about something wonderful that somebody has done. This whole thing has shook me like nothing I can remember. But I'm proud of everyone involved with Slashdot for working together to keep a line of communication open for a lot of people during a crisis. I'm not kidding myself by thinking that what we did is as important as participating in the rescue effort, but I think our contribution was still important. And thanks to the countless readers who have written me over the last few days to thank us for providing them with what, for many, was their only source of news during this whole thing. And thanks to the whole team who made it happen. I'm proud of all of you.
A request
(Score:4, Insightful)(https://sourceforge.net/projects/ruoppolo/ | Last Journal: Monday December 12, @03:51AM)
Thank you
Well Said
(Score:5, Insightful)(http://www.bloodshed.org/)
During my life I've always taken "bow your head and pray" as "shut up and look serious".
And I thought Slashdot did a very good job this week. I woulda emailed Taco that, but I figured there was enough traffic over the Internet.
Really good job guys. Between Slashdot and Drudge I felt as informed as a guy can be.
Re:A request
(Score:4, Interesting)(http://www.clueology.com/)
I almost wonder if this is a subconscious attempt from people to rally support for white Christian America. It's easy for those who fit comfortably into the majority to see this as an Us vs. Them battle.
Those people, however, are hideous hypocrites; They are "Christian" in the same way that those flying the suicide planes were "Muslim"; i.e., only in that they happen to use that term to describe themselves. Any white Christian who starts seeing those of other ethnicities or religions as "Them" is not only a poor excuse for a Christian, but ought to be considered as bad as the terrorists themselves.
Re:A request
(Score:5, Insightful)(http://members.verizon.net/~mbuckley)
Here [washingtonpost.com] are some such people.
Re:A request
(Score:5, Insightful)(http://www.wota.com/)
A statement like this is meant to strike fear in the hearts of those people he "fingered" or people who may associate with them. Isn't that the core definition of Terrorism...
Shame on you Falwell, for you are no better than those responsible for the thousands of deaths on Tuesday.
Re:A request
(Score:4, Interesting)Re:A request
(Score:5, Insightful)This reminds why I told my wife I don't like organized religion. I told her "I doubt you'll ever hear a statement like: an extremist atheist bin-laden follower today killed himself and 100's of others". She just didn't get it. She doesn't see how the hatred of the so called christians like the Pat Robertson's of the world can take us down the same path as the Taliban and the Bin Laden's.
Remember it was the senior President Bush that said there is no place in America for atheists. I say, yes lets get the people behind the attacks, but don't destroy the very foundation of our democracy in the process.
Re:A request
(Score:5, Interesting)(http://rimbosity.com/ | Last Journal: Saturday September 27, @03:24AM)
Speaking as a white Christian...bingo. You just hit the nail on the head. In fact, it's that very attitude that allowed these terrorists to believe that what they were doing was somehow God's Will.
Christianity, Judaism, and Islam are all filled with references to people who, though they weren't Official Churchgoing Believers, represented God's will better than the average Believer. And an ongoing theme in both the Talmud and the Bible (I can't speak for the Koran, although I've been told Mohammed's teachings are very tolerant of other religions) is the failure of church leaders.
It's ironic. All of these religions which these misguided fundamentalist-whacko "leaders" (such as Osama Bin Laden and Jerry Falwell) supposedly follow condemn the most the Bin Ladens and Falwells of the world, who use God's Name to mislead people, or cause people to commit terrible atrocities.
Christianity is at its worst...
(Score:5, Interesting)(http://jean.nu/)
I won't go into a long diatribe at the offensiveness of this mindset or this behavior, but rather reference it in order to point out that, as a genre of religion which is bent on conversion, i.e. selling their viewpoint to others, Christian sects tend to be obsessed with appearance as much as substance. Whether it is cloaked as "setting a good example to others," "representing your faith/church to others," or "demonstrating through actions what it is to be a good Christian," none of which are as blatent as the Mormon adage of "avoid the appearance of evil," the underlying message is clear: appearances are at least as important as substance. With a mindset like that, reinforced every sunday from one's spiritual leaders, is it any suprise that people who look even a little non-mainstream garner the reactions like you describe?
We should kick ass and eradicate our enemies. Not in the name of God, not in the name of some religion, but in the name our our country and our people, which have been attacked and shall be avenged. Keep church and state where they belong, separate, and obliterate the bastards who committed these atrocities last Tuesday in the name of our secular, democratic instutions, leaving each of us to pray, and to grieve, in our own fashion, according to our own beliefs. And never make the mistake that just because someone doesn't share your beliefs, ethnic background, or skin color that they are in any respect less capable of grieving than you.
Pray Or Meditate Or Whatever For President Bush
(Score:5, Insightful)Correction
(Score:4, Informative)(http://slashdot.org/)
"I'm a loving guy. And I am also someone, however, who's got a job to do and I intend to do it. And this is a terrible moment," Bush said.
http://www.foxnews.com/story/0,2933,34322,00.html [foxnews.com]
I hope you're wrong about the nuke.
Re:You are going to be scared....
(Score:4, Insightful)(http://captionthis.com/)
In Executive Orders, terrorists flew a plane into the White House, killing most of the major gov't officials that were there for a ceremony. Sound famaliar?
Don't remember which book it is (Never finished it), but there's one about a small terrorist group gathering materials and machining a nuclear bomb to set off in a football stadium in Colorado. The possibility of it seemed to be almost zero...but so did the possibility of a hijacked plane being flown into the Pentagon. What's scary is that it's not as hard as it seems. With bin Laden's resources, he could have procured the materials and gotten people to carry a nuke-in-a-briefcase into the WTC, and taken out all of Manhattan. Not that I'm saying what happened isn't terrible - it is - but the possibilities for terrorists these days are WAY past frightening.
Terrorists don't care about diplomacy, they don't care about the politics, they care about causing pain, terror, death, and destruction. How are we supposed to fight an enemy that is free to use weaspons many times more destructive than our government will dare to use? How can you guarantee safty?
Look at me...I'm turning into a paranoid conspiracy theorist.
Re:A request
(Score:5, Insightful)(http://slashdot.org/)
As an atheist who hangs out with a lot of other atheists, let me say that this is a strongly filtered perception. It is filtered by the fact that most of the time, an atheist isn't going to really say or do anything that would clue you in to the fact that he's an atheist. It's not a religion - it's not a belief. It's a lack thereof. So it rarely comes up in conversation unless in response to something someone has said that is highly insulting to an atheist (for example the 700 club quote in this thread). So you never realize someone is an atheist until they've been riled up by such a statement and are (justifiably) argumentative about it. This gives the false impression that atheists aren't compassionate toward other religions. It's not that we aren't compassionate. It's just that we don't tend to say much about it when we are in that mood. Consider today's prayers across the country, and the "moment of silence" that is typically used for personal prayers. Think about whichever crowd you were in at the time, or were watching on TV. During that moment of silence, if everyone in the crowd was an atheist - would you have even noticed the difference? Not really. It was a moment to respect the dead, not to start arguments.
I suspect you are committing the statistical error of counting the hits and ignoring the misses. My compassion for a religious person dissapears when he tells me I am incapable of being a good person because I'm an atheist. But until then, I view him as a person just like any other.
Over the years I've noticed that religions are so varied and complex that they don't *really* dictate what morals people believe in, like they claim to do. Rather, what happens is the other way around: that the morals a person holds dear determines which religion, and which subsect of that religion he migrates toward, and this is mostly subconsious. You can twist a lot of religious teachings into whatever you like by selectively picking which parts to take literally vs which parts to take as "metaphor". Thus when someone tells me he's a Christian, or he's a Muslim, or he's a Jew - I try as hard as I can to *not* draw any predjudiced conclusions about what that really means. I wait to observe his actions and his words first. Or at least I *try* to. I will admit to not always succeeding at being gracious and polite as I should be - especially if I've just recently heard some hateful rhetoric such as that of today's 700 club (which has been floating around the net all day).
As a child I was brought up as a Ba'hii (Not sure where the apostrophe is supposed to go there - it's one of those Arabic words that just isn't easy to spell in English letters.) They are a modern offshoot from Islam, in much the same way that Christianity started as an offshoot of Judaism. (But the Ba'hii offshoot is much more recent - from the 1800's). Anyway, one of the core tenets of the Ba'hii faith is that one must spend time studying the other religions of the world as well as one's own, because they all came from the same god, and all have useful things to say. Given enough time studying, one is supposed to realize that the relgions all teach the same message, and that the similarities between them must be the "pure" message from God and the differences are supposed to be the minor points that don't matter as much, perhaps are even manglings of the original message a bit. It's a nice idea, and Ba'hiis tend to be *extremely* tolerant of other religions because of this. It is common to mix up different words from different relgions in a 'reading'. (The televised event today from the National Cathedral, with the list of different leaders from different faiths speaking is very reminiscent of what I remember from Ba'hii services I used to go to.) Anyway, where am I going with this? I don't really know.
I guess just one day I started toying idly with the idea that maybe the reason the religions overlap so much in what they say isn't because they all come from the same god, but because they don't come from a god at all, and instead come from people's own hearts and desires. Perhaps their similarities merely reflect the similarities cultures have because we're all human, and at some basic level we all think similarly when confronted with the taks of explaining the unknown. Over the years that little seed of doubt grew until finally I started thinking it the more plausable explanation to the one offered by the Bah'ii faith. When I turned 15 (the age at which one can officially make the declaration to become a Bahii for real), that doubt was so strong that I didn't feel comfortable doing it, and so I didn't. And I never have since, and what was once a small doubt is now what I see as the most plausable explanation. I called myself "agnostic" and decried atheists until much later when I realized they don't actually *have* a belief like I thought they did. I was applying the atheist label to a strawman that doesn't often exist. That's when I started calling myself one too.
But enough rambling. On this day, we shouldn't argue. Comfort yourself with the things you hold dear. I often think of religion as just a crutch, a teddy bear to make people feel better. But today is the last day to be taking people's teddy bears away. They need them now more than ever.
Re:A request
(Score:5, Interesting)(http://www.stingdesign.com/)
Speaking of religious fanatics, we have our own here in the US:
God Gave U.S. 'What We Deserve,' Falwell Says [washingtonpost.com]
Jerry Falwell and Pat Robertson blaming the events on liberals, feminists, etc. etc. etc.
Sick.Time for some highly unpopular opinion...
(Score:5, Insightful)(http://www.mixdown.ca/)
Jerry Falwell and Pat Robertson blaming the events on liberals, feminists, etc. etc. etc.
While I wholly condemn the actions of the terrorists I do have to critically ask, "Did the government of the United States of America have this coming?
You'd have to be blind to see that the U.S. government has been supplying arms and training and money to factions around the world for over 50 years. You'd have to be blind to see the American government change its mind mid-stride -- first by supporting a group (again, with weapons and money), then by turning face, cutting off support or even condemning the actions of the group they supported.
You'd have to be insane to believe the 1973 crap propaganda article by Gordon Sinclair is a clear and frank view of the United States of America and its leaders and their policies.
The government of the United States of America has been bullying and harassing nations for a very long time, flaunting themselves as a superpower which is untouchable. They've stuck their noses in other nations' business too many times and someone had decided to cut it off.
I don't agree entirely with this Guardian article [guardian.co.uk] but it does rise a very strong and important point: The U.S. must change the way it carries itself in foreign affairs. The American people must stand up and take active interest in their nation's government. The American media must stop downplaying foreign affairs.
an aside: the Canadian people aren't much (any) better in this regard. Canadian readers: How much interest do you show in your government??
I do not believe that this is the act of one nation, or even of a nation. And I am frightened because I do not think this is the last.
The U.S. government and media is running around crying "Why me? Why us?" and you have the President standing frail and shaken, telling his nation that "He's gotta do what he's gotta do" instead of analyzing the situation properly and keeping cool.
I must give Bush credit -- he did not spout off about Arabs or "them guys" as Clinton did with OKC -- Bush remained calm and rational. I fear that this is quickly fizzling out because his anger is taking over and as President, he is not allowed to have those emotions. He is a man with the power of a very large, wealthy and military nation. He is not allowed to be angry. I think he is grappling with those emotions and his reserve is failing.
As a Canadian, I demand retribution for what happened in the United States this week. I am not saying "forgive and forget." Blood will be shed, and rightly so. Check out my /. userpage [slashdot.org] for views on what I personally feel is acceptable for retaliation. I also think the President should send a strong message that it is not acceptable to hate the middle eastern people -- Just as there was no witchhunt against all white people with OKC, there should be no anger towards the Arab, Muslim and other middle-eastern people within or outside the U.S. This is not an attack by the middle eastern people nor their religion; this is an attack by terrorists and cowards too cowardly to stand up and fight.
And I fear that we will be brought into a world war because of it.
Re:Time for some highly unpopular opinion...
(Score:4, Informative)(http://www.artboy.org/)
I barely know where to begin when I read crap like this. The simple truth is that people hate us because we're the biggest kid on the block.
yes, they care about individual policy decisions, but there isn't a nation on earth that doesn't make the exact same decisions every day. WE're criticized for butting our noses into foreign affairs, then criticized for being isolationist if we DON'T get involved in foreign affairs.
We're criticized for supporting side A against B, but if we switch sides, we're criticized for supporting B against A.
There is no policy we could possibly have that would make other nations happy with us. If we withdraw, we're "ignoring our responsibilities" but if we get involved we're "flaunting our power".
Well fuck you all very much, planet earth. We didn't ask to be the only superpower. We're not itching to feed your hungry or shelter your homeless or finance your economic devastations, but we're the ones you call on first when you need those things done.
You complain because american hegemony is destroying your cultures, then you go out and buy coca-cola and watch Friends on TV. You complain about our imperialism while ignoring the fact that Germany and Japan are our biggest competitors exactly BECAUSE we rebuilt them at OUR EXPENSE after we could have conquered them.
We're damned if we do and damned if we don't, so don't give me any shit that we had it coming because of our policies. NO FUCKING POLICY WILL MAKE EVERYONE HAPPY!
We're like the prettiest girl at a party -- all the women want to be her, all the men want to fuck her. There is not a country on earth that wouldn't trade places with us in a second, and on days like today i'd almost be happy to do it.
Re:Time for some highly unpopular opinion...
(Score:4, Informative)(Last Journal: Tuesday October 22, @01:56AM)
yes, they care about individual policy decisions, but there isn't a nation on earth that doesn't make the exact same decisions every day. WE're criticized for butting our noses into foreign affairs, then criticized for being isolationist if we DON'T get involved in foreign affairs.
There's some degree of truth to your statement -- not all the reasons america is disliked are legitimate, and the prettiest girl at the party metaphor was great. We've done some great things in the world.
But indications are that we've been responsible for some pretty terrible things at times in our nations history. We've supported and trained regimes who've used terrorism and torture because it suited us. Israel is a prime example. Several central american governements are an example.
There's a book called "What Uncle Sam Really Wants" by Noam Chomsky with some details. I'm still trying to figure out how much of it to swallow, but he paints a grim picture, and if even a fraction of it is true, we have a lot to own up to here in the US.
That said, attacking such a large civilian target as the WTC in peacetime is unprecedented and completely wrong. The people who did/would do that need to be fought.
Re:Time for some highly unpopular opinion...
(Score:4, Interesting)(http://www.mixdown.ca/)
I barely know where to begin when I read crap like this. The simple truth is that people hate us because we're the biggest kid on the block.
That is not entirely true. I simply don't believe it's possible to convince people to die terrorizing a country because they're the "biggest kid on the block." The hatred runs far far deeper than this and it must be resolved if there is any hope of this not happenning in the future.
Did you read the Guardian article? Have you looked into your country's foreign policies over the past 50 years? There are people who have very legitimate reasons to be upset at the U.S. because of the way it handles itself in foreign lands.
Well fuck you all very much, planet earth. We didn't ask to be the only superpower. We're not itching to feed your hungry or shelter your homeless or finance your economic devastations, but we're the ones you call on first when you need those things done.
Please go do some research before you post so you don't come off so totally ignorant of the reality.
Actually yes the U.S. does strive to be the only superpower. What did America do to Russia? What is it doing with China? I can't believe you actually think that nations don't strive to be the biggest and best. I believe that the U.S. is the only nation not paying its UN membership fees. Or behaving poorly in UN. Or trying to be the police of the world. The U.S. government does flaunt its power; that is what I believe got the United States into this mess in the first place!
You complain because american hegemony is destroying your cultures, then you go out and buy coca-cola and watch Friends on TV. You complain about our imperialism while ignoring the fact that Germany and Japan are our biggest competitors exactly BECAUSE we rebuilt them at OUR EXPENSE after we could have conquered them.
Actually I don't drink Coke and dislike Friends, but thanks for generalizing.
Your comments on WW2 are laughable at best. The American government sold arms (to both sides? I don't remember) when WW2 broke out. America didn't come to the aid of the other nations; it waited until it was hit before getting involved, instead profiting from the war. Whether this reserved attitude is good or bad I reserve judgement -- but don't think that for a second that America saved the world; if you read through your history you'll see that the Soviets and British played key roles in ending the war, as did America. It was a group effort.
We're damned if we do and damned if we don't, so don't give me any shit that we had it coming because of our policies. NO FUCKING POLICY WILL MAKE EVERYONE HAPPY!
You're right, America can't make everyone happy. Instead of meddling in the affairs of foreign nations by selling arms and supporting conflict, it should do more peacekeeping and put its own military into action. The "hands off" kind of "support" that the U.S. has done in the near past reminds me of a parent shouting at fighting kids to stop instead of getting their hands dirty and stopping the conflict themselves.
Or, better yet, let the nations battle amongst themselves without making profit from it! Maybe even by providing relief to the civilians!
As a citizen, you can stand up, take notice and get involved in your government and the politics of your government! What was voter turnout this past election, something like 46%? While Canada isn't much better in this regard, we don't have the problems that America has and our foreign policies seem to be less convoluted and more focussed on peace than profit.
I hope you realize that I did not say the American citizens had it coming. Killing thousands of civilians is horrible and will be met with vengeance. Terrorist attacks like this are beyond words. No war was declared, no "plain-to-see" warnings... this is terrorism of the worst sort.
But to think for a second that America is right and innocent and that they didn't have an attack coming is just plain stupid, ignorant and totally unbelievable. I am sorry it happened and I do mourn the loss of civilian life but I am also not blinded by my patriotism like I believe you are.