dead sun's Journal: New Moderation Thoughts

Okay, so I've made a little noise now, and I think this is my initial description of how I would do moderation of a message board. I've put a decent amount of thought into it, but would appreciate comments and questions as well as suggestions for refinement.

There are two basic premises. One is that everybody has an unlimited number of moderations. The second is that there are other people will moderate similarly to you.

Amazon uses the type of system I am in favor of to make recommendations for you to buy DVDs, games, whatever. It is quite effective.

So let me run over some things. The moderation system would be on a scale of 1 to 5 stars, from tripe to great. It's a very simple system that can be put into a database rather easily. It has the downside of being rather flat, not distinguishing between being insightful, funny, flamebait, or whatever else. However, looking at /. it's rare to see those used properly anyway. For this aspect we keep things simple. This is a reversal from my initial thought that was aiming for big multidimensional vectors of attributes, because it is too little gain for too much complexity.

As stated earlier in the post, everybody has unlimited moderations. You can set any post at any value from 1 to 5 stars, or leave it unrated. You can do this any time to as many posts as you want. Everybody else can too.

Now, I don't have the details currently on how to make this sort of query fast, but here is the overall idea behind it. I know a couple people in the U of M's GroupLens/MovieLens group that does this sort of thing on a decently large scale, so I can probably get some tips. Anyway, on a periodic basis (nightly, weekly...) a table which lists a relative strength of agreement on posts between users is updated. This data is calculated from where users moderate the same posts. If I moderate a post the same way you do the strength goes up, opposite and it goes down.

This table has the possibility to not only be very sparse but also very large (users^2 cells). At a byte per cell it's roughly 250GB for 500,000 users. Not exactly pretty. However, I believe there are easy ways around it since our problem isn't exactly recommendation but something similar. If each user has their own table which includes the top 100 (1000?) other users that have moderated the same posts as they have, then the size shrinks drastically to a linear growth by users without that much effect on moderation performance. Even the top list is easy enough to calculate by a simple joining and counting query. That also makes it easy to get rid of a particular user should they decide to leave.

With this information in hand a user views a page of comments. Each of these comments is then scored based upon the moderation of that comment by others in the user's list in magnitude of the typical agreement of the users (even oppositely).

First, a little note, 1 star should equate to a value of -2, 5 stars to a value of 2. Simple shift. Now, lets do a small example. I'm reading and have a list of 5 users which frequently moderate the same posts as I do. Here's the table:

User : Agreement

A : .8
B : .5
C : -.2
D : .3
E : .7

So, I find which of the users have moderated this particular post. I then take the absolute value of each and sum them up. Then I divide each agreement in my list by that value. Then I take that list of normalized agreements, restore their sign, and multiply their rating of the post times their respective value.

Let's pretend that A, B, and C moderated this particular post with values of 4, 5, and 2 stars, or real values of 1, 2, and -1 respectively. Then we have a total sum for normalization of ABS(.8) + ABS(.5) + ABS(-.2) = 1.5. This makes the amount of influence from each .53, .33, and .13 respectively. This is multiplied by their post rating to give .53 * 1 + .33 * 2 + -.13 * -1 = 1.32. Rounded out it equals 1, which translates to 4 stars.

Now just imagine that on a larger scale.

How to calculate agreement. A simple ((4 - ABS(my rating - your rating)) / 4) gives an agreement on a single post. Average those for all posts moderated by both parties and you have agreement. This could be a big job so is done periodically and stored.

Why will this work? Because moderation is all relative, none of it is absolute. Nobody can get modded into the ground unfairly by just a couple people, nor can anybody be modded up unfairly by just a couple people. It might be possible to get a big block of people to collude and try to break things, but then it's just their group they break it for. If it continues then they get posts moderated in opposition more often and they lose the power to influence others. Though no longer required, it necessitates tracking moderations. Abuse is out.

The downsides are a little complexity, though I doubt this is too complex. Also is that people have to actively moderate things to improve their recommendation style moderation. I have more ideas on this below. Finally there is server power. It will require more juice to do this than to simply track the current score between 1 and 5.

So about the active moderation bit. I have two suggestions. The first is to have new users get an average moderation to all posts displayed. The new users get this until they've moderated 50-100 (rough numbers) posts themselves. This gives the system enough time to try to seed the user's preferences. The other is some sort of reward. Use the 10, 25, 50 scale of rewards with different color stars next to their name or something. People love that sort of thing. So at 0-9 moderations no star, 10-24 moderations a yellow star, 25-49 a green star, 50-99 a orange star, 100-249 a blue star, etc. A change at every 1 * 10^x, 2.5 * 10^x, and 5 * 10^x.

So those are my very long winded ideas on moderation. It isn't super simple, but it isn't a horrible cess pool of overly complex crud either. What do you think?

Wow...

[FortKnox]

Very complex but definately nice moderation system. Looks like it'd work quite well. I only worry slightly about implementation.

It will work with my opinion (wanted 5 different moderation 'levels' and all), just adding the 'see recomended moderation' thing to it.

How would a user not logged in see the moderations? A combo of everyone??

Re:Wow...

[dead sun]

I think a not logged in user could see the average of everybody's moderations. This has potential for abuse in that a sufficiently large group of people could moderate everything opposite what other people are moderating, bringing the average close to 3 or fewer stars, neutral to negative ground. This could be done by a person browsing two windows, one logged in and one not. With unlimited moderations they just mod opposite the consensus, or just down for everything. Enough accounts doing this in contrast t

Re:Wow...

[FortKnox]

You know java?

Re:Wow...

[dead sun]

To a certain degree yes. I'm more of a C++ or Lisp person, but Java is similar enough in syntax to C++ that I'm comfortable enough with it.

Just don't expect me to do any fancy Java specific stuff without a bit of time for sorting through reference. It's a language I've been meaning to do more in though, and would be happy to do a first shot at anything and have somebody look over it for ways to make it more Java optimized or specific.