Update (5/28/03): The information in this journal is outdated and no longer reflects the state of Slashcode; this journal is a historical record but is no longer accurate.
--
Slashdot Moderation : Exercising Agreement
A journey into the statistical methods employed to create Comments That Agree .
As I discussed in my previous journal entry, nearly 500 moderators were manually and permanently removed from the pool of potential moderators for giving positive moderation to this comment. While the manual banning of potential moderators by the Editorial staff is not documented, the source code used to accomplish this is readily available and the practice appears to be quite commonplace. The reasonable questions that arise from this practice are: how many moderators are being banned, what is the objective, and what are the actual results? The Slashdot Editorial staff has repeatedly stated that they don't have time to answer these questions (they're certainly not addressed in the FAQ), so it's up to us, the readers, to ponder them.
How many moderators are being permanently banned from the moderation pool? No one knows for certain except perhaps the Editors. Let's look at what we do know. Five hundred moderators were banned for moderating this comment. If we assume that one such comment is posted on a daily basis, that is approximately 185,000 potential moderators banned per year. That post was probably an exception, so we must revise our number. Jamie McCarthy made a post to a User-Created discussion which is deleted every two weeks, so I must reproduce his comment by cut & paste:
"Well, again, we have the logs, so we know what we're talking about. We log both how many times we give mod points, and how many times we would have given mod points but didn't. In the former category: 99.73%; in the latter, 0.27%." - Jamie McCarthy, Slashdot Editor
Additionally, Rob Malda made a related comment in a User-Created discussion:
There are a quarter of a million daily readers, but (and I just checked) only 1.2% of them post. So understand that those of you posting in the forums are already a minority of the community. - Rob Malda, Slashdot Founder & Editor
Now we have some numbers to run with. Of the 250,000 people who read Slashdot, 3,000 of them have an "Interest Level" high enough to post comments here. This "Interest Level" stands at 1.2%, or .012. Now that we have the "Interest Level" for comments, we can use it to determine the "Interest Level" for Moderating & Metamoderating. Moderation requires less personal risk, so let's assume (to be as conservative as possible) that far more people want to moderate comments than post them. Let's assume that up to five times as many people want to moderate comments. There are probably far fewer, but since we are dealing with an assumption, we should make it a conservative one. This makes the "Interest Level" for Moderation about 6%, or .06. Now, Jamie has stated that 99.73% of readers receive moderator points and are not banned. Their "Interest Level" is .06, which means that (99.73 *.06 = 5.98) 5.98% of these people (at most) actually use their points. Now we have the banned moderators, the .27% that Jamie mentioned, who have been banned for bad moderation. Banned moderators have an interest level of 100%, because they have all used their points to get banned by an Editor. So, now we have a figure for the number of Moderators and Metamoderators who are included and excluded by the manual intervention of the Editors. (.27/(.27+5.98)) = .045, or 4.5%, is the lower bound for the percentage of moderators being manually excluded from the pool. If you run the numbers again assuming that the Interest Level for Moderation is the same as the Interest Level for posting comments, you arrive at a figure of 18.4% of moderators being manually banned. This would be our conservative upper bound. So, depending on how interested people are in moderation, between 5 and 18 percent of moderators have been manually banned from participating by the Editors.
What is the objective of the Editorial staff in manually removing User Moderators? That's a really tricky question to answer. Keep in mind that the Editors have gone to extensive lengths not to answer that question, so we must assume that it is an emotionally loaded subject. However, we can interpret some of their actions to arrive at an answer. The post mentioned in the beginning of this article was repeatedly moderated as Offtopic by many Editors. Rob Malda described it this way:
however only a smaller percentage of those moderators actually use their points... hundreds of users moderated it up [and I subsequently banned them all] I modded it [the comment] down a few times too. - Rob Malda, Slashdot Founder & Editor
Note that text in brackets was added by myself to reflect what we've learned. Reading that statement makes it pretty obvious that the purpose of removing Moderators from the pool is to ensure that the results of the Moderation system are consistent with what Rob Malda believes that the results of the moderation system should be. Moderators are chosen from a pool of Slashdot users who have not moderated contrary to the wishes of the Editors in the past. As the Slashdot FAQ states, "The Slashdot Editors have unlimited mod points ... These moderations represent approximately 8% of all moderation". In short, nearly ten percent of Moderation is accomplished manually by the Editorial staff, and the rest of User Moderation is accomplished by users that have not disagreed with the Editors about how to Moderate. What does this mean?
When surveying the impact of the Moderation system, there is one overridingly important statistic. That statistic is that over 99% of Slashdot readers do not post comments. Keep in mind that the default view of Slashdot has a threshold of one. This means that if you're moderated to zero or less, 99% of the people who would have read your comment... won't. Books could be written about whether or not blocking communication to 99% of the possible audience constitutes "Censorship", whatever that means today, but for the purposes of this essay, the point is moot. Suffice it to say that Moderation determines 99% of visibility, statistically. Now, consider that the Slashdot system, as a whole, is a constantly evolving system in which Moderators can transition from unbanned to banned, but not vice versa. Moderators who moderate differently than the "guide" Moderations of the Editors (which constitute 10% of the Moderation) are removed from the pool of influence. Therefore, Slashdot as a system moves in only one direction: towards promoting comments that are Moderated in agreement with the Editors. What is the role of Metamoderation? Let's look at the FAQ:
according to Meta Moderation, the fairness of these [editor Moderations] is statistically indistinguishable from the moderation of non admin users (92-93% of moderations are ruled 'Fair').
Keep in mind that the 5 to 18 percent of Slashdot Moderators who are banned from participating in Moderation are also banned from participating in MetaModeration. The purpose of MetaModeration, therefore, is that of reinforcing agreement. Those who have agreed with the Editors in the past are allowed to vote on whether the Editors moderate fairly. Not surprisingly, this subset of potential MetaModerators often agrees with the Editors. If they didn't, they'd probably have been removed already! MetaModeration, therefore, serves to reinforce agreement with the Editors among the selected Moderator pool, and Moderation serves to enact agreement with the Editors among the comments visible to 99% of Slashdot's readership. We only need one more quote from the FAQ to understand what this means:
Goals [of Moderation]: 1.Promote quality, discourage crap.
Since the Moderation system is a self-reinforcing system that promotes comments that agree with the Editors, we must assume that this comment means two things:
- What the editors believe is quality.
- What people who disagree with the Editors believe is crap.
That may be a strongly worded value judgement, but it stands as a conclusion easily & objectively reached. But aside from this value judgement, Slashdot stands as an example of a best-of-breed solution for those who wish to create a community that promotes agreement. Slash appears to have been specifically designed for this purpose, and it can be employed to create a community that agrees about.. just about anything! For instance, ask a Slashdot reader if Linux is a great Operating System. The answer will almost certainly be an emphatic yes! Rob Malda has created an excellent system for managing a userbase into agreement, and for the most part, the users agree with the system. Which was probably the whole point.
Slashdot: News for Nerds, Stuff that Matters. Comments that Agree.
