glh's Journal: The slashdot eff- is it getting worse, and what can be done?

Ok, I know this topic has come up SEVERAL times (especially lately), but I just have to rant. It seems that "slashdotting" (ie, effectively DDOS'ing other sites because of the large spike in hits when a story is posted) is getting WORSE, or maybe I'm just noticing it more.

I posted a reply the other day on the subject and it generated a lot of discussion (as well as trolls/flames). Basically it seems like the main argument that people use is the one in the FAQ about "how hard it would be" (in terms of legality/permission) and "other sites might get mad because of lost revenue". Well, google caches and I believe the only thing they look at is robots.txt. The revenue issue seems moot- if a site is down, what revenue is it going to get??! On top of that, what about the increased bandwidth? So, it doesn't seem like it is THAT big of a deal. I think it would be a nice service to the community, both slashdot and otherwise. Few things are more irritating than having an article posted with a link to a site that doesn't work.

Any ideas for how those of us fed up can be heard (I've already counted 3 slashdotted articles today.)? I'm thinking about doing an "ask slashdot", but I'm sure it'd just raise the usual "RTFM" trolls and whatnot. Maybe we can do something similar to the T(H)GSB that happened a while ago (not sure if that made any difference).

I say we just deal with it

[D_Fresh]

Yes, it's annoying and inconvenient, and prevents you from being able to make an informed comment on a story. But it's also an inevitable result of the kind of web we've built, with unequal bandwidth allocated to the sites. Slashdot isn't a problem with the system so much as it's an exception to the rule, which is that sites that need higher bandwidth generally purchase it, while the obscure sites that don't purchase it do so because they really don't need it. Slashdot breaks this rule by pointing up otherwise obscure sites to a lot of people very rapidly. Some people might say there's responsibility on the CmdrTaco/CowboyNeal end to make sure that the sites they post are able to handle the onslaught, but that's kind of ludicrous. In the end, I think it's just a reality we have to deal with, and in truth I don't encounter it so much that it makes Slashdot unreadable. Just a minor nuisance that usually resolves itself in short order (i.e. sites are mirrored, articles are posted as comments, or sites go into hibernation for a bit until the initial wave passes).

Bottom line - the Slashdot effect has that name because not every site can create a DDOS attack so spontaneously and innocently. It's an exceptional test of an otherwise resilient system.

Re:I say we just deal with it

[glh]

Bottom line - the Slashdot effect has that name because not every site can create a DDOS attack so spontaneously and innocently. It's an exceptional test of an otherwise resilient system.

That's a good point, and the reality of it. And I agree- it isn't really the responsibility of Slashdot (though some may disagree), but it would be a nice gesture and helpful.

Heh, some might say "beggers can't be choosers" especially when it comes to features like this. But then again, this is Slashdot!

I'm not giving up just yet, but I think you definitely have some good points. Hopefully taco will come around....

Temporal caching

[Osty]

CmdrTaco gets close to the perfect solution in his FAQ entry (in the FAQ [slashdot.org]), yet doesn't seem to realize it. That solution? A temporal cache only used on smaller sites. The FAQ says:

So perhaps we could draw the line at sites that don't have ads. They are, after all, much more likely to buckle under the pressure of all those unexpected hits. But what happens if I cache the site, and they update themselves? Once again, I'm transmitting data that I shouldn't be, only this time my cache is out of date!

So, don't cache the sites that can afford the bandwidth and don't need to be cached (though the supposition that ad-supported sites can afford the bandwidth is unique to the time period in which that FAQ was answered, and probably doesn't hold as true now). Instead, cache those smaller sites for a limited period of time (say, 6 hours?), after which the slashdot-local cache becomes nothing more than a redirector to the original site (perhaps get fancy, and if that site goes away revert back to the cached copy, so that older stories can still be useful).

On a different note, there is one line in that answer that does really get me, and it's this:

I could try asking permission, but do you want to wait 6 hours for a cool breaking story while we wait for permission to link someone?

Funny! What's 6 hours, when Slashdot has already missed the story by a day or more, or is reposting an earlier story? Rob tries to make it appear that Slashdot is on-the-ball with their stories, but any regular here knows that just isn't the case. (Okay, if you only rely on Slashdot for your daily intake of news and interesting stories, then it doesn't matter if the story is 6 hours late or two weeks late -- you're not going to see it anywhere else. For the other 99% of us, we've already seen probably at least 50% of the stories elsewhere.)

Re:Temporal caching

[Osty]

I forgot one other point I meant to add in favor of temporal caching. The stories on sites that would benefit from it generally don't change frequently. Certainly not in a fairly short period of time (6 or 12 hours, maybe a day -- long enough for the story to cycle off the front page). In fact, the only time I've seen such a story change is when the site blocks referrals from Slashdot with a note saying as much. In that case, I'd think it doubly important to cache the site, rather than make readers have to do the legwork themselves to ferret out the correct URL and visit there directly rather than just following a link.