Slashdot Posting Bug Infuriates Haggard Admins

Last night we crossed over 16,777,216 comments in the database. The wise amongst you might note that this number is 2^24, or in MySQLese an unsigned mediumint. Unfortunately, like 5 years ago we changed our primary keys in the comment table to unsigned int (32 bits, or 4.1 billion) but neglected to change the index that handles parents. We're awesome! Fixing is a simple ALTER TABLE statement... but on a table that is 16 million rows long, our system will take 3+ hours to do it, during which time there can be no posting. So today, we're disabling threading and will enable it again later tonight. Sorry for the inconvenience. We shall flog ourselves appropriately. Update: 11/10 12:52 GMT by J : It's fixed.

Only one thing to say

*Clap clap clap*

So does a first post ...

... roll over to be a last post?

D'oh.

Anyone could have made the mistake.. good to keep us all in the loop though :)

And let this be a reminder to the kids - RTFM, twice!

I for one

welcome our 2 to the power of X overlords.

Last post!

Last post!

who we should REALLY blame

Alright, who's the joker who posted the 16,777,216th comment?

Thanks for breaking slashdot, jerk :D

Oh Noes!

Its like y2k, only worse!

Congrats taco

Does this mean that comment id#16777215 has the longest thread in history?

Can anyone actually find it to see - I tried but could only get to 16777217 [slashdot.org], its likely to be in a journal or just a reply to an older article.

I sense a disturbance in the force

As if a thousand geeks all made the same damn "last post!" joke at once. . . . . .

Better make it longer

I mean, look how quick we got to 16M comments. 4.1 Gigacomments will come in hardly any time at all. I predict we'll be doing all this again in merely a few weeks!

So who's the killer?

I wonder who posted comment #16777216. That person should win some sort of "I borked Slashdot!" award.

Re: Oh Noes!

"it's like y2k but worse"

I know what you mean. Y2K was supposed to put an end to civilization, but at least we'd have been able to post on slashdot.

seems strange

...why wasn't this problem discovered on the dev system in advance?

It's true that...

Anyone could have made the mistake

But it's so much funnier when that anyone is not you.

Check out...

...comment 16777215 [slashdot.org].
Mmmm... CT, are you sure the parent index was your only problem?

2^24

"2^24 comments ought to be enough for anyone" -- CmdrTaco

Comment 16,777,216 does not exist

Some of you are asking which comment it was that got the cid 16,777,216. The answer is that none did. For redundancy, Slashdot is now running multiple-master replication which skips values for auto-increment [mysql.com]. Our db-1 assigns odd-numbered primary key IDs, and db-2 assigns even-numbered. Right now writes are going to db-1 so newly created rows will have only odd IDs.

The comment that got 2**24-1 was this one [slashdot.org], if anyone cares :)

Sorry about the inconvenience, everyone.

Digg?

That's cool, I'll just pretend I'm on Digg, with its 1981 Commodore 64 BBS-style threading.

  Wait..sorry Commodore fans. I know it had better threading than Digg.

Haggard?

Haggard admins? Does this mean that the Admins will go buy some meth and get a massage?

My Reply to the Funny Comment Above This

Uh, this is a reply to the 8th post down from the top (remember to use this like an array and zero reference). Yes, I'm talking to you, admdrew.

You claim that the 16,777,216th comment would have broke it but I contest that actually the 16,777,217th comment poster would be the culprit. Since it should be able to handle that many comments if it is zero referenced, and it would actually be the one after that one that would break it. You laugh but these kinds of problems plague a lot of coders?

If you don't agree with me, please respond below and reference my comment ID.

16 million posts ...

... should have been enough for anyone.

Holy Chit!

No threading? Welcome to Farkdot.

Re:OMG

Posting comments is my entire life!! What will I do?? :-(

I know how you feel; reading them is my life, too.

And?

Sorry for the inconvenience. We shall flog ourselves appropriately.

And post the YouTube link?

It's a joke, kids

Dupe! I TOTALLY posted this story like, last WEEK man! (I laugh, but I betcha someone might post this in seriousness)

Did somebody say...

Flogging and Haggard in the same sentence? If we can get "crystal meth" in, we'll hit the trifecta!

Sounds Familiar

I used to work at Comair. Remember, that airline that stranded about 10,000 people in the airport a couple of Christmases ago? Same deal. Program was capable of handling only a certain number of changes. Hopefully your president won't have to resign.

So was it fixed in Slash?

So is the bug still in the CVS revision of Slash, or was it fixed 5 years ago and Slashdot never applied the patch?

Give Slashdot Subscriptions to the borkers

Give a 2^0-year Slashdot subscription to the guy who hit the limit and one to the the first non-administrator guy who successfully posted after the fix.

If you can find the first guy who COULDN'T reply due to the limit, give him one too. He deserves something for his trouble.

EldavoJohn cid=16786251 reply

Actually, comment 16,777,217 couldn't break it, because that comment's parent cid could have only been 16,777,215. Up until then, there wouldn't have been an overflow value put into the db.

There's no telling which comment it is, because (16,777,217 + 2n) might not have been a reply, meaning it would come up correctly.

Dorks.

Does this mean that Slashdot is going to denegrate into Digg now?

Reply to 16786251

Reply to comment number 16786251 [slashdot.org]:

``You claim that the 16,777,216th comment would have broke it but I contest that actually the 16,777,217th comment poster would be the culprit. Since it should be able to handle that many comments if it is zero referenced'' ...but it's probably not zero-referenced. Typically, ids in SQL start at 1.

Today is the first day I've had to type subjects.

@eldavojohn, #16786251 [slashdot.org] (god, this feels like digg now)

Uh, this is a reply to the 8th post down from the top (remember to use this like an array and zero reference). Yes, I'm talking to you, admdrew.

You claim that the 16,777,216th comment would have broke it but I contest that actually the 16,777,217th comment poster would be the culprit. Since it should be able to handle that many comments if it is zero referenced, and it would actually be the one after that one that would break it. You laugh but these kinds of problems plague a lot of coders?

If you don't agree with me, please respond below and reference my comment ID.

I certainly admit I wasn't thinking 0-based when I wrote that. The question is, though, should we blame the person who wrote the last valid comment (therefore ruining the fun for the rest of us), or whoever wrote the first broken comment?

Also, is everyone going to add the obligatory 'parent' link on their posts today?


[ Parent [slashdot.org] ] - [ Reply to this [slashdot.org] ]

Access to the Database?

Any thoughts on making the DB publicly accessable other than through teh Dot? Not sure what I'd do with all that data, but I'm sure these's a grad student somewhere who'd love the opportunity...

May i be the first to say...

Brillant!

I always wondered where Paula Bean ended up...

RE: 16777217 GET

mod parent up

Mod parent up

Uhhhh who's your daddy?

Why are all 16 million+ comments in a single table

Slashdot being a news (for nerds) site, I would expect that the usage patterns are such that a huge majority of the content accessed by users is very recent -- say, perhaps, 90% of the database hits are for stories and comments that were posted in the last week.

So why, pray, is this usage pattern not accounted for in the database design?

MOD parent up!

Mod parent ... wait..

This was fortold a few months ago...

http://slashdot.org/~themusicgod1/journal/137880 [slashdot.org] ...ok, so it was obvious...

Take all the time you need

Take all the time you need, I'm more than willing to refrain from posting durin.... Oh shit!

Trolls

At least nobody can feed the trolls now!

Slashed Eyeballs

If Slashdot released the Slashcode more frequently, with more/better comments/docs, and encouraged some of the many of us who complain about bugs/features to help the project, then it's more likely that someone would have debugged this bug earlier.

Open source - it's not just a buzzword, it's a way of life.

A real Slashdotter!

And this is why you should not have arbitrary limits in your programs, ladies and gentlemen. Not even limits on the values your numbers can represent

Now this is a real Slashdotter! This guy knows how to build an infinite computer!

24 bit?

Why on earth does MySQL have a 24 bit integer datatype? On what platform does it even remotely make sense to use that in the first place? It's going to get cast to 32 bits for any arithmetic operations anyway, and on most platforms today alignment requirements are going to pad the extra byte in memory and disk, so you're not even saving any space. Why even give someone the option over choosing between 16 bit and 32 bit integers?

Re: Why are all 16 million+ comments in a single t

poot_rootbeer asks why all the comments are in one table, when the data access pattern is such that 90% of our hits are on only the most recent entries in that table.

The answer is that we used to do it this way but it's a huge pain. In 2000 we converted from having two tables for 'stories', recent and archived, and merged them together. The performance hit was not big, and it made the code so much simpler it was a no-brainer.

It's the database's job to cache properly whether we split the table or not, and the database does that just fine. The only performance problem could be when there is a rush of inserts, or updates to the same sets of rows, spanning both newer and older portions of the table, and that just doesn't happen.

If we did want to do this we wouldn't split the tables manually; the code complexity is too high a price to pay. In MySQL 5.0 we would use a MERGE [mysql.com] engine, which has issues of its own but would involve smaller changes to our code. That's still not worth it for us. What we're probably going to do is wait for MySQL 5.1 to get out of beta and then do some performance testing on tables partitioned [mysql.com] by date and see if that gains us anything. For example, a SELECT on our comments table could be limited with a WHERE clause to only retrieve rows with a date >= the discussion object's date, which for 90% of our queries MySQL 5.1 could optimize to only look at the most recent partition. If the gains turn out to be significant, then since partitioning involves very limited code changes, we'll probably do that. Generally speaking, though, database performance is not a problem for us. So far our main bottlenecks have been CPU and RAM on the webheads. As long as we don't do anything stupid our database performance has been fine, though, as today proves, we are quite capable of being stupid.

[ Parent [slashdot.org] ]

Re: Access to the database

Any thoughts on making the DB publicly accessable other than through teh Dot? Not sure what I'd do with all that data, but I'm sure these's a grad student somewhere who'd love the opportunity...

Not just grad students; as a DBA by profession, I'd love a crack at the DB. If nothing else, it would give me a great place to play around with MySQL. Not to mention the ability to maybe extract some interesting user-level statistics.

Of course, the odds of this happening are pretty damn low - there'd have to be an awful lot of work and review done to scrub the DB of information that is entrusted to /. that people didn't plan on having released to the internet at large. Passwords, for example (even if they're stored only as hashes, getting the whole DB would make it feasible to crack them); real email addresses, real names...I assume that the subscription process doesn't involve actually storing credit card information in the DB (I don't know; I've only used PayPal), but that might be another concern.

Just the email addresses would be a huge deal - can you imagine the market value of such a targeted list of addresses?

In short, it would be fantastically cool for them to release the DB, but it would be a lot of work on their part for no particular return. Not to mention that if they released it once, they'd no doubt be pestered to keep releasing periodic updates...then there's the bandwidth issues...and, even, the potential copyright issues (/. doesn't own the copyright on posted comments, the poster does)...then the copyright issues for stuff they do own; releasing the DB would make it trivial for a bad actor to post a mirrored slash. A little bit of domain typosquatting and some ad deals, and you could be talking about real money.

If I were them, there's no way in hell I'd even think about doing it.

But it would be cool.

Parent [slashdot.org]

Let the flamewars begin...

Would not have happened if Slashdot used PostgreSQL.

Let the flamewars begin...

Unthreaded flame wars are much less enjoyable.

graphic artists

The number 2^24 is of interest to digital computer artists, as that is the number of unique colors combined in the commonly implemented "True Color" RGB8 space. That color space is looking pretty limiting in some respects, but that is truly a lot of unique colors when you think about it. A 16 megapixel image does not need to repeat any color used.

If all slashdot posts from the history of Slashdot were sorted into color bins,

• every post including the -2 trolls would get their own unique color,
• all the colors which are predominantly blue would be claimed as "first posts,"
• all the colors which are predominantly green would be unfunny memes like "hot grits,"
• all the colors which are predominantly red would be complaints about editing or journalism,
• the pure cyan, magenta, yellow shades are moderated as insightful or interesting,
• the 256 posts corresponding to completely neutral gray shades are actually insightful or interesting

Once that were done, people could simply post their replies as a reference to existing posts. "Hey, #938D3A to you, buddy!" "Know what I think of that? #F2C2A9!"

Who needs data integrity?

Ah, MySQL. Where trying to insert a row with a column value larger than the column can actually store results in MySQL clipping it to the max value.

mysql> create table test (field mediumint unsigned);
Query OK, 0 rows affected (0.05 sec)

mysql> insert into test values (20000000000);
Query OK, 1 row affected, 1 warning (0.02 sec)

mysql> select * from test;
+----------+
| field |
+----------+
| 16777215 |
+----------+
1 row in set (0.00 sec)

Fortunately, as of MySQL 5, you can fix this problem.

mysql> set session sql_mode='TRADITIONAL';
Query OK, 0 rows affected (0.00 sec)

mysql> insert into test values (20000000000);
ERROR 1264 (22003): Out of range value adjusted for column 'field' at row 1

So, yes, early versions of MySQL had a brain-dead default SQL mode that simply "corrected" invalid column values, but MySQL 5 fixes this.

Now if only they would add column constraints...

Finally, a tale for my grandkids in the new era

One day, Cmdr Taco is designing his database, and he sits down at a table with three integers on it. First, he tries the baby bear's integer, but exclaims "2 meager bytes is way too small for my appetite."

Next, he tries Papa bear's integer, but proclaims "4 bytes is way too big for my little site, I'd just end up wasting so much."

Finally, he tries Mama bear's integer, and extols "3 bytes is just right," not noticing it was really the same as Papa Bear's bowl in disquise.

/. copyright question

Get your mod points ready, this is off topic, but considering the current state of discussion anyway, I don't feel so bad about it.

Regardless, while writing this post [slashdot.org] regarding why the /. admins won't (and shouldn't) consider releasing a copy of the /. DB to the public, something occurred to me.

Comments on /. are owned by the poster, according to that one line that shows up on all the comment pages (specifically, "The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.") At the same time, though, /. doesn't provide a method for having comments you've made removed from the DB.

If I own the copyright on the comments I've made, shouldn't I be able to rescind publication rights on them, and prevent /. from displaying them in future? Or is there some kind of implicit license in posting on /.? Did I clicksign an agreement covering this when I joined (this was getting on towards a decade ago, so I really don't remember the joining process at all)?

Or are publication rights, once granted, irrevocable?

Of course, I suppose asking questions when there's no way for people to hit reply is a specific form of vague insanity...still, I'm curious.

So how do I get to post #1?

I'd really like to see it. I bet it goes something like, "what's this stupid web thingy anyway? I bet it'll never make it to version 2.0..."

Re:Old discussions

Sorry, I think index was a bad choice of words.

Whenever a post is made and it has a parent cid, that number must be stored in the table.

If MySQL saturates instead of rolling over (see this comment [slashdot.org]), then all replies after comment 16,777,215 will have the wrong parent cid, and I don't think there's any way to fix it.

+10

Ten points for honesty!

At least they didn’t try to make bullshit excuses. I respect them for being up front about the real nature of the issue.

"flogging definition"

As your punishment, you should write some kind of data-mining algorithm that starts from the point you disabled threading and try to construct intelligent threads based on the subject and the body of comments...

Use MySQL and eventually it'll bite your arse

I guess this is another thing to add to the MySQL gotchas page [sql-info.de]. Of course, in a decent database engine, like PostgreSQL, if you alter a column data type then the indexes are updated to reflect this.

Re: /. copyright question

In reply to this [slashdot.org]:

Or are publication rights, once granted, irrevocable?

IMHO, probably. When you published a comment, there was no mechanism available to you, for removing comments, or even the slightest hint that one might appear in the future. You knew what you were getting into, so it could be argued that you accepted what happened.

In a more general sense, it is well-known that sending a packet to the 'Net is like opening a bottle and letting a genie out. Once you do it, it's out, and if you ever manage to shove it back inside, it'll be more a matter of luck, rather than something possible because of policy.

Re: graphic artists

If all slashdot posts from the history of Slashdot were sorted into color bins, ...

So, from a distance the site would blur into an ugly pastiche resembling lime green?

So why was this 24-bit to begin with?

With all teh funnae posts about it, let me be the first one to ask: why were you using 3-byte integers to begin with? Why would anybody anywhere ever use these for any reason at all? What advantage to these have? Why was this table laid out like this? This doesn't make sense to me at all. Were you really imagining that shaving a byte off each post was going to save you DB space? I can't quite believe that. But than what exactly would be the motivation for using such an odd integer size?