Forgot your password?
typodupeerror
User Journal

FortKnox's Journal: Static Comment Pages? 16

Journal by FortKnox
I've got a theory:
In order to speed up Slashdot and not use up as much processing power, they are batching up the comments, and spitting out static pages every few seconds.
Normally, the page is dynamically created when a person views it.
The reason I say this is because twice I've posted to an article, reloaded the page the article was on, cleared my cache and reloaded again, and I didn't see my post. After a few seconds, it miracuously appeared.

Its a clever idea, it definately will speed up the site, but it will also create more redundant posts, because people don't see what they want to say (because its batched up for the next static page), so they post it. I guess its a money vs redundant content thing...

Thoughts?
This discussion has been archived. No new comments can be posted.

Static Comment Pages?

Comments Filter:
  • I wonder how many monkies thay have at the zoo here?

    oh, you meant about the static pages.

    It seems to be working ok, I've never really run into problems. And if it spees up the servers, then great. If people keep posting things twice, well, that's what -1 redundant is for.

    • Are what SlashDot is for. Nobody reads all the comments.

      And certainly not many moderators bother with redundants. I know I don't... Not that it would take a lot of effort - it just seems like a waste to try to get rid of a mountain with a shovel.

      SlashDot is just too big to be an effective discussion site.
      • SlashDot is just too big to be an effective discussion site.

        Agreed. I used to try an read comments, post my thoughts and such. Now, there are way to many crap comments and most people don't want to hold a discussion after they hit submit. I miss the old days of newsgroups where some threads would span days.
  • You weren't behind a caching proxy or anything when you tested this, were you? My work does that...

    A much better design would be, instead of caching pages for every individual, caching a single discussion object, then forwarding that to the client side for processing. It would be a pretty substantial change, though...
    • FWIW, I hit the same thing yesterday and I've never seen it before.

      It is a good idea, though, if that's what's going on. I'm amazed at how long they've been able to serve up dynamic content, as the readership grows and the scoring and display keep getting more complex.

  • It depends upon the level that you go. They have the static HTML pages that get dumped out, but I still don't understand this.

    Slashdot doesn't get that much traffic, especially on their comments. I mean, it gets a lot of traffic, but we're not talking Yahoo/Amazon traffic.

    My mod_perl framework that loads about 5K lines of code and has an internal caching system can handle 39 requests a second on a P2-400. I think Slashdot's problems all go from a bad codebase.

    They've patched and patched but it probably is just making it worse. The static dump of the comments (and it will even say, "This comment may take a few moments to show up on the static pages") will only be for the popular views as well. If you have modifiers setup for your karma bonuses and stuff, it will dynamically generate the page.

    As I try to post this and slashdot ceased responding...
    • One of the biggest issues is the amount of SQL queries done - a heavily nested discussion thread could result in over a hundred SQL queries.

      A site like Fark.com with flat discussion threads has it much easier - just do something like SELECT * FROM comments WHERE article_id = 20394 and you're done. Much less nasty for the SQL server to handle.

      Comparing Slashdot and Amazon/Yahoo! is hardly fair, too. :-p
      • One of the biggest issues is the amount of SQL queries done - a heavily nested discussion thread could result in over a hundred SQL queries.


        That is bad scoping. If you have a hundred SQL queries, you should be shot. If there are 900 queries, why not select them all? It doesn't take much CPU power to write a nesting algorithm using hashes, probably only 4x that of a flat discussion. You can even remove things that are 2 levels down and reduce it...

        Slashdot is just bad design.
        • From my teachings, if this was the case, you should just have a stored procedure or a view to give you everything in one select. DB's are made to be ultra-fast on queries, so you should always give the searching to the DBs, not try to do all the processing in the program (not to mention that DB's are usually on seperate machines in enterprise apps, meaning you are paralleling the processing).

          Things like doing a 'sort' in perl vs an 'order by' in SQL are simple things that would drastically increase speed.
          • From my teachings, if this was the case, you should just have a stored procedure or a view to give you everything in one select. DB's are made to be ultra-fast on queries, so you should always give the searching to the DBs, not try to do all the processing in the program (not to mention that DB's are usually on seperate machines in enterprise apps, meaning you are paralleling the processing).


            SQL doesn't work in a heirarchical structure though, so there is no way to properly nest things, at least not in a data structure that could be immediately exported to XML.

            A proper design of a threaded discussion system is using stored procs and putting as much ordering in the database process as possible, but just format the results using a nested algorithm.

            It's actually pretty easy and I've written several that work very well. You go from the bottom up, and store them in an array of comment ID's without parents, then add children to the parents and have a hash that stores the traversed list. I can post some sample PHP code that doesn't have DB optimizations that I whipped up just to work.
            • SQL doesn't work in a heirarchical structure though, so there is no way to properly nest things, at least not in a data structure that could be immediately exported to XML.

              Actually, a lot of that depends on the RDBMS you are working with (in terms of heriarchical support). For instance, Oracle has "connect by" which is very efficient in exploding hierarchies, even more so than a stored proc.

              I'd prefer to use the DB to store it. In my mind, parent/child is a data relationship and code shouldn't have to figure it out, it should just be responsible for manipulating or displaying it.
              • I'd prefer to use the DB to store it. In my mind, parent/child is a data relationship and code shouldn't have to figure it out, it should just be responsible for manipulating or displaying it.

                True, but even Oracle couldn't handle doing something like an HTML formatted nested comment list. You can pull a heirarchy out, but not have a comment that links to another comment.

                When I get around to porting all my threading discussion code to mod_perl (Go MagicBox development Go) I'll probably abstract the queries to use the strong points of whatever database I have access to. Oracle, Postgres, MySQL, etc.
  • I've suspected the same thing and if you look, you'll see that that "feature" utterly screwed over the Carmack fuel debate, for one. Redundant posts really are a different level of problem when there are four or five times as many of them.
    How many of you even noticed that Carmack wrote two long, detailed posts and at least one short one? Given that they showed up on the second page within minutes of being posted, probably not many.
    This may be a "cool" bit of programming, but if it doubles or triples the work to find useful data in a thred, as this most certainly can, then I for one would rather have the old system back.
    I certainly understand that /. is under appalling cost pressures but this is not the way to go. Much tho I hate to say it, maybe it's time for /. to institute some nominal fee (say, three dollars a month) for people who want to have journals and otherwise rise above lurker and AC status.
    Speaking as a content guy (tech, edit, and business) I simply don't understand the model that /. is operating under. They've got hordes of possible ways to generate revenue and don't seem to be pursuing any of them. Given how long /. (odd though it may seem) has been a key venue, there really should be ways to sell access to searchable archives in ways that don't require selling content per se. I am utterly f*cking broke, but even I would shell out five bucks or so a month for full access to the archives. That means ALL of a user's comments, not just the most recent 24. It means searching for content, not just in heds (which are useless). It means some simple list of threds going back to the old days, or at least a friggin' year.
    I know that this sound very un/., but do you folks realize how useful such an interface would be for folks at places like Stamford and MIT where people now get degrees in the history of technology? For cryin' out loud, the book reviews and Ask Slashdot's alone are a damned goldmine of information. Much better for some things then Lexis-Nexis, for which journalists, lawyers, and such pay scads of money.
    Add a premium level with (oh, no! blasphemy!) moderated comments to weed out the Goatse.x filler and the value would skyrocket. Add the means to charge per search and it's all good. Think how great it would be to be able to bring up all the book reviews on, say, a programming language you've been meaning to get into. Would that be worth a fifty cent micropayment?
    Speaking of which, the folks who tried to do this sort of thing back in the dot com mania days concluded that two things really kept them from being viable. The first was getting enough content into the system (yeah, like /. needs more content) and the other was micropayment systems. Well, guess what? It's been YEARS since those conclusions and there are starting to be decent systems out there (finally!). Western Union is getting serious, a bunch of dudes out of MIT have something coming out. Blah, blah blah.
    Yo! Cowboy! Taco! Dudes! I'll spec out the whole damned implementation gratis if you'll give me an account and a credit for having done it.
    Of course, for all we know, this is all already in the works, but since, as has been pointed out elsewhere, the great and mysterious sages of /. don't condescend to even acknowledge that slashlag and the rest are even going on, how can we tell?
    Anyway, color me peeved. I really hope that the /. folk get their act together soon, but I utterly do not believe that the key is to add toys while cutting costs and meanwhile still make no sincere efforts to make money. It's not a goddamned hobby site anymore. And as long as it keeps being treated like one it will keep getting less stable and usable.

    Rustin
  • twice I've posted to an article, reloaded the page the article was on, cleared my cache and reloaded again, and I didn't see my post. After a few seconds, it miracuously appeared.

    What you saw can be explained by /. holding new posts in a queue pending processing by the lameness filter. They have always said there will be a short delay befoe your post goes up.

    As to your theory - I don't hink it would be workable given there are so many viewer preferences now where the sort and selection is very dependent on all the +/- multipliers, your friends/fans, etc. Each person has to have their own page generated - and the could store these cached pages for every single users currently reading a thread. Just the R/W process of the cahcing would be as much work as generating a new page when and if you asked for it.

    They do use the system you talk about for the main section of the main pages though.

  • When a story is posted it's showing no comments. Then if I post something, suddenly there's 150+ comments but I can't read any of them. This place sucks, yet I can't leave.

"Call immediately. Time is running out. We both need to do something monstrous before we die." -- Message from Ralph Steadman to Hunter Thompson

Working...