Slashdot Moderation Phase 1.1

As many of you have noticed, something has changed recently with Slashdot moderation. Its not my major change that is still coming, but instead its a significant tweak to the old system. Hit the link below to read the summary of these changes, and comments about them. This is extremely important to any of you who read or post comments.

Back in the old days, I was the only moderator. Dave pitched in. It was easy, there were only a hundred comments each day. We read them all. They were almost always good- only a couple were really flamebait or offensive, and those were just deleted. This got sticky- I never liked deleting comments.

As Slashdot grew to have hundreds of comments each day, I coded a fancy moderator system. One that didn't involve deleting comments, but rather just organizing them for people who wanted it. Over the following months, friends, family, pets, and even a few bots tried to do the job. At the end we had 25 people moderating a total of about 5 comments per day. It was obviously not working- any regular reader of the comments knows that the signal to noise ratio is simply terrible in those flamebait articles. The system grew to have thousands of comments each day, and the 25 moderators (of whom only a 2-5 were active on any given day) simply couldn't keep up.

I have a new system that I'm devising, but I've decided to experiment with the old system. I think I can learn from it, and make the new system better. Or else, this "Tweak" might even work and then I won't have to write the code. We'll have to see.

So what is the change? Simple, we had 20 or so moderators. Now we have 408.

What?

Yup.

The system was tracking moderation done to each user internally for just this purpose. (I had a score of 2, Anonymous Coward had a -1628 *grin*) All users with a positive score were given moderator access.

Last week wed. we had 2,800 comments posted on Slashdot. 11 of them were moderated. This week 15-30% of comments are being moderated, and its my hope that this number will increase.

Now a lot of you guys are going to scream and cry about censorship, but that just isn't the case. Anyone can disable the actions of the moderators by simply setting their default user preference to -5 or something really low. Tada! Slashdot in all its flamey off topic glory. But my goal is that users reading with a preference set to 3 will only read the absolute best comments. That type of reader doesn't want a discussion. They don't care about the 300 comments- they just want those 2 comments that are really smart, insightful, and often, better then the story that they are attached to. Try setting your Comment Limit to 10, and your Comment Order by Score. Suddenly the few comments that you see are interesting. They're useful.

The goal here is to create a better dispersal of scores. Last week, a +4 comment was virtually impossible, but we've had 40 since the new system took place. Sure, not all of them were great, but as a whole, they were good comments.

Now the danger. With 25 moderators, it was pretty easy to keep an eye on things. But with 400 its going to get simply crazy. We're going to have abusers. I've already revoked access from a few people. For you moderators, read those The Moderator Guidelines carefully. The rest of you might be interested too. The general summary, is the moderators shouldn't let their own opinions factor in. They do and thats the problem. Its my hope that since we have 400 of them, we'll have some abusers (who will hopefully surface and have their access removed) but they'll be outnumbered by honest, fair people who don't let their own ideals interfere with the task at hand. Its a difficult task, but an essential one.

A few of the more important rules for moderators:

• Impartiality. This isn't "I agree with That", this is "That is worth reading, and that isn't". This is obviously the hardest, and most subjective part of the task, and the one that will require the sharpest eye on everyone's part.
• Anonymity. Any moderator who posts that they are a moderator will probably have their access revoked. I simply don't want moderation to be an ego thing.
• Accountability. Anyone who sees clear breaking of the above 2 rules should send me info (I need a URL to the comment: cid & sid. Click the reply button and send me that URL if you need it). This isn't "3 Strikes and Your Out". If someone is abusing their trust, they'll lose it.

As an aside, if you have problems, bugs, or complaints, email them to me. I don't read all the comments. We have 2800+ of them on a good day. There's no way I'm gonna read them all. Send problems to me. Posting complaints is usually off-topic. Emailing me is much more likely to get a response, plus if you want to complain about how much I suck, don't do it in a story about CD Vending Machines or Wearable PCs- its simply off topic. Do it in this story! Its on topic here. Or email me so I'm sure to read it and cry.

Where is this heading? Think of a news site like Slashdot without a guy like me, or a group of guys at the center. One where the best comments become the articles on the homepage. If we could make that work... wow. At some point I'll have a page of the top 10 comments from the last 24 hours. I think that will be really interesting- I'll probably have a general discussion at some point specifically for this purpose.

Its a delicate thing trying to make all 75,000 of you happy- Your tastes are diverse, and there's just no pleasing all of you 100%. So I'll keep trying new things, and make as many things customizable as possible, so most of us can have it the way they want it.

We're getting closer. But until then, hang in there. Constructive criticism is appreciated (although I simply can't reply to everyone) I even read the flames, although if you make me cry I don't reply.

Update: 03/23 01:53 by CT : Responses to some of the comments:

1. No, simply creating new accounts won't work. You had to have had a comment moderated up by the original 25 moderators.
2. No, moderators can't moderate their own comments.
3. I yanked someone already for revealing that they had access. Someone didn't read very carefully.
4. An absolute minimum for comments? Set it to -10000 or something. I doubt we'll ever see a comment that bad *grin*
5. I'll probably figure out a clean way to reassign moderator access occasionally. I haven't thought that far ahead yet.

Questionable Moderator Motives (ATTN ROB)

While I can appreciate the desire to try and form some order out of chaos, this whole moderator thing sits very ill with me.

Why? Because I've had my posts moderated before, in some cases, severely.

I'm not talking about "First Post" or "THiS SuX" or profanity-filled invective or other such stuff that I think we can all agree falls firmly into the "noise" department - I'm talking about on topic, carefully written commentary.

I've gone through some of these posts, trying to keep an open mind and objective viewpoint, looking to discover what it was that was so objectionable to require moderation - and I found nothing.

My only conclusion is that the moderator in question disagreed with the content of the post, and moderated it in an attempt to supress the viewpoint.

I don't consider myself all that radical, but yet I've been moderated. If this has happened to me, then surely it has happened to others.

This is wrong. It's Evil.

Worst of all, I was never given a chance to "face my accuser". Not ONCE have I learned WHY the particular post was moderated.

Rob, there should be some sort of "moderation history list" tagged to every comment. Each downward moderation should have a REASON tagged onto it, and the identity of the moderator should be listed - even if that identity is just a moderator number, untraceable by anyone other than you back to the meatspace person.

Like this:

[18:02-03/22/1999] Downgraded to -2 by #402: Offensive language

At least this gives us victims a way to track who is doing what and for what reasons to our posts.

DG

Moderation - Silly

Thats exactly how it works. Except only 400 people see those little buttons. I'm trying to keep tabs on it so that people don't push idealogies, and instead are fair and impartial. We'll see if it works.
Rob "CmdrTaco" Malda
Pants are Optional

Continuing my previous comment...

Ooops... I pressed the submit the submit button too soon.

2. It might encourage other (more sensible) /. readers to post sensible comments because they would feel that their comments would not be drowned out by the noise of non-sensical ravings and rantings.

I do have some questions however:
-- Who guards the guardians? -- How are the moderators evaluated? I don't know if there is a mechanism by which we can identify and evaluate how the moderators are doing their work...

I might have a suggestion though:
-- Identify the moderator who gave the rating -- not by their user name but something like the id that telephone operators give out so that we can complain to their supervisors but not know their real name...
-- Allow a search of articles/comments by moderator id so that everyone (not just CmdrTaco) can evaluate what they do and complain if they show a distinct bias or not...

Also some minor nitpicks:
1. comment count on main page should be based on comment level a user selected.
2. comment spillover should also be based on comment count that reflects the comment level a user selected.

Other than that, I like it very,very much!!!!

Won't scale - tragedy of the commons

I'm afraid I disagree with both of your ideas.

Let's start with the use of real names. There are several reasons why this is a bad idea, and only one actually decent one: accountability. Admittedly, some folks would be less likely to post offensive or just plain stupid things if their own real name was attached to it for all to see, and that is a plus. They might also be less likely to post unpopular or controversial views, and that is the biggest weakness. Other people really value their privacy a great deal, and wouldn't use their real name to post *anything* much less something juicy or passionate, and they would be a great loss, too. And some people just don't use their real names much on the net, so using it doesn't make a lot of sense anyway. I knew a guy in college who always went by his nick, doubt. I happened to know that this was derived from his real name, and even knew what it was, but nobody used it. If there was some post with his actual parent-given monniker, I'd have no idea who that guy was. But if it said 'doubt' I think, hmm, oh yeah, I know him. I use reemul on the net. I'm not trying to hide behind some pseudonym, I just like the name. Anyone who wants to track down all my real-life info is welcome to mosey on over to internic and lookup the contact info on my domain, listed above. I think it even has my phone number.

Next is cost. Your (I feel) arbitrary $29 figure might have some utility in filtering the participants in Slashdot, in the sense that it may limit users to only those who feel that Slashdot is important enough to pay the money. This would certainly have the effect of drastically diminishing the numbers of casual users, folks who only wander by from time-to-time and post rarely if at all. But is that a good thing? Perhaps a person attracted to the site by word-of-mouth or a link in some other article has something valuable to say, or is at least amusing. Why cut them off? And certainly I doubt that this would cut down on trolls and flames. If I pony up the cash, I get to say whatever I want. Killing my access means that Rob would have to give me a refund, with all the touch costs and handling fees and problems with the credit card billing company. Ugh. So we may get less folks, but perhaps less inhibited folks. And what about people for whom $29 is a hardship? The students, the struggling freelancers, the international readers on the down side of the exchange rate? I wouldn't want to lose them, either. On the other side are folks for whom $29 isn't really that much at all, who could pay it and still not really care about the site. I blow that much on my lunch break buying a book and a nice meal most every weekday, $29 doesn't indicate commitment on my part.

(And just for you conspiracy theorists, do you think that the all those Microsoft employees you suspect are posting will be deterred by a measly little $29 fee?)

So, while I understand your concerns, I don't think that your ideas are the way to go.

-reemul

Fixing unfair negative scores

One thing that should be pointed out - not all the unfair negative scores are intentional. I just noticed a comment, Score=1, that while not deserving to be stricken from the page didn't deserve to be filtered up, either. So, I hit the checkbox to knock it's score back down a point.

Apparantly somebody else agreed with me, and tried to fix the problem at the same time, because when I came back, the comment was gone completely, Score=-1.

Rob, a request: Could you set all Moderator thresholds to -100, and just let those who really want to change them back? One of the most important ways to spend a moderator point is fixing an unfairly negative-scored post, but I suspect 98% of the moderators still have their thresholds set not to see negative-scored posts at all.

A Brief Defense of Meept

Every castle needs its Court Jester. Like it or not, "MEEPT" is our Jester.

He/she/it should be given an unchangeable status of +99999, provided the priviledge only be abused in ways that would be fairly and democratically irksome and/or distressing to ALL SlashDot readers accross the board. This would provide much needed comic relief in times of strife and quite possibly bring about an end to World Hunger and Y2K paranoia.

Humbly,

Skip Kent

Several Questions, Some Comments, and Ideas

#end_lurk_mode
Lurk mode off.

99% of previous comments unread due to extremely slow link (I apologize if I cover something previously posted)

Comment: This is probably one of the best moderation systems I've seen so far.

Question 1: Is there a heirarchy of moderation, i.e. are there "Super" moderators who can change/revoke moderation that another moderator has done, remove moderator status from lower moderators, etc?

Question 2: Is there some sort of trial period (please don't mention a duration or specifics) for new/added moderators?

Question 3: Is there/will there be a system for moderators to provide feedback to comment posters as to why their comment was moderated (i.e. "flamebait", "MEEPT!", etc.)? This would be good to help those of us who think we're writing good comments, yet we're either stating the obvious or are off-topic but don't know it.

Question 4: Now that my brainstorm is getting underway, with all the recent customization, how about an area for "trusted" "part-time" story posters? Like those of us who have a story on Linux that's more than the "Linux on CNN" stuff, but almost never gets posted, e.g. the BeroLinux-Linux Mandrake distro merger? Something where people can turn off part-time story posters if they choose, and maybe combine the moderation feature into that so part-time posters are rated on the importance/quality of stories they post. To me at least, this seems like something that could provide more news (especially on those slow days) and still leave the "original" Slashdot for those who want it.

Question 5: How about a way to alert/flag stuff for moderators to moderate? Like a MEEPT! post (sorry to pick on MEEPT, but that's the only example I can think of right now) could get flagged for moderator attention, or something that goes waaaaaaay off-topic. Sounds useful.

Question 6: Is there some sort of "real" democratic system for the suggestion of new features and other feedback? I mean, is there like a congress of moderators or people who vote aye or nay on stuff after reviewing it, and their vote counts so many points, and higher-ups can veto it, etc.?

Question 7 (rhetorical): Will this actually be read and responded to, and acheive a high score? ;-)

---
I'll gladly take full responsibility for this comment if you'll gladly mail [e-mail] telling me why you like/dislike it.
James R. Turinsky, LilithFair.Org/SomeSites.Com owner/administrator/Linux user and advocate.

Why not just set up an NNTP server?

I know there's probably some really obvious point I'm missing, but I don't understand why /. doesn't just switch to an NNTP interface. *Particularly* with 75,000 users, and with most articles getting 200+ comments within the first couple of days.

Using NNTP, people could use whatever newsreader suits them best, and do their own filtering and/or scoring the usual way.

There could be a slashdot.headlines group, for instance, where all articles are (cross-?)posted, with followups set to their relevant sub-topic groups: slashdot.debian, s.redhat, s.microsoft, s.tech, and so on. Moderate the s.headlines group to ensure that it contains *only* the headlines, and not followup discussions, and that's pretty much all that would be needed, it seems.

One more time...with FEELING!

Okay people...lets look at some reality here:

Fact is, the moderation system that Rob's coming up with is prob the best that could really be asked for.

It's basically peer-review, which in a world with people making comments varying from "first post" to the recipe of coca-cola, is probably the best possible form of censorship available (sorry Rob, but any limiting of comments is a form of censorship, though not necessarily in a bad way)
If you want the mindless blabber, then set your preferences low, if you want meat and potatoes, have some higher standards...very simple. Granted, the anonymous coward postings are lower...WONDERFUL! We don't need fingerpointing, but some credability is a great idea, IMHO. And even though there is a lot of good stuff that comes from AC posts, they'll get adjusted as moderators read them and up their status...and *POOF* they appear on my comment list too! amazing, isn't it?

This way of doing things lets the people that actually might contribute something useful help decide what the comparatively useful comments might be. THIS IS A GOOD THING!

Rob: not to kiss ass or anything, but fact is, you've done a great job handling the sh*t we feed slashdot sometimes, and I'm glad you're doing your best to look at your 'baby' as the community that it has become...great job...and on the same note, letting or helping us rule ourselves, well...welcome to democracy...:o)

Good job rob...(and David...and anyone else that fails to be mentioned)...:o)

BTW...When do we get to write the new constitution? ;o)

Suggestion: Moderating the moderators?

Although direct democracy moderation might sound like a good idea in theory, it would probably not work well in practice - imagine a KDE vs. GNOME flamewar in an article's comments if everyone were able to moderate. The scores would have as bad a signal-to-noise ratio as the comments. So a specific group of moderators provides a useful buffer.

But moderators must have accountability too, and if there are a lot of them, Rob will have a hard time managing this himself.

So how about this suggestion:

* Make article scores floating point.

* Give each moderator a weight, which is the amount by which he can change an article's score
when moderating. Initialize all moderators to 1.0 and make 1.0 the maximum (or pick other suitable parameters).

* Let everyone vote on whether or not they like the way an article was scored. This would feed back to the weight of the moderators who moderated it towards it's current score.

Thus, ultimately moderators who consistently score articles up or down for bad reasons will have their weight lowered until their moderation does not really affect anything.

This sort of applies the principles of representative democracy to moderation: moderation will be insulated from the momentary whims of the masses, but ultimately in a long-term sense, moderation is under the control of the entire community.