How to Stop Digg-cheating, Forever

The following was written by frequent Slashdot editorial contributor Bennett Haselton. He writes "Recently author Annalee Newitz created a bit of a stir with the revelation that she had bought her way to the front page of the story-ranking site Digg. Since Digg allows any registered user to go to a story's URL and "digg it" in order to push it upward through the story-ranking system, it was inevitable that services like User/Submitter would come along, where a Digg user can pay for other users to cast votes to push their story up to the top. User/Submitter says they are currently backlogged and not taking new orders, but they say the service will return and will soon feature services for manipulating similar sites like Digg competitor reddit. Even if the new U/S features are vaporware, it probably won't be long before other companies offer similar services. But it seems like all of these story-ranking sites could prevent the manipulation by making one simple change to their voting algorithm."

Before getting to that though, what's at stake? The revelation that Digg could be trivially manipulated did not cause the site to be overrun with bogus stories all at once -- most of the links on the front page still look interesting. Newitz said that her story, which was deliberately chosen to be as lame as possible, got buried by users soon after it hit the front page, which is how Digg cleans spam stories out of the system. However, she also said that in the time that the story was on the front page, the story got about 35,000 hits, whereupon her server crashed and the traffic was thereafter divided with two other mirror sites; presumably if the server had stayed up, she would have gotten about 100,000 hits, all for an initial expenditure of $100, which is orders of magnitude cheaper than buying advertising any other way. (If she had done the same thing with a good story instead of a deliberately lame one, presumably the traffic gains resulting from word-of-mouth and repeat visitors would have been even higher.) As long as the benefits outweigh the cost, more and more unscrupulous users are likely to pay for such services, and since the service provided by User/Submitter is easy to copy, probably similar services will spring up to drive the price down even further. If nothing changes, then eventually sites like Digg and reddit will be flooded with nothing but paid stories. Most of the stories on the front page will probably still be interesting (why would you pay to promote a link, unless it was good enough to draw repeat visitors and get the most value for your money?), but everybody who didn't pay for votes would eventually get crowded out.

One Good Samaritan, Jim Messenger, managed to shut down one Digg manipulation service called Spike The Vote, by buying it out (for a paltry $1,275 - they must have wanted to get out fast) and then turning over to Digg. He warned people that the moral was: Don't sign up for Digg manipulation services, since Digg might get your information from them and then you'll be banned. Actually, I think the moral is simpler: if you're going to try anything like that, do it from a throwaway account that you don't care about losing if you get caught. (Or, only sign up with manipulation services which publish a privacy policy promising never to share your information, especially not with sites like Digg. Then if Digg buys them out, then the site has violated their privacy policy and Digg as the new owner inherits the liability for that, so you can sue them, right?) But as the idea spreads, it will probably become impractical to play whack-a-mole by shutting down manipulation services as they keep springing up. Any time the cost of providing a service (clicking on a few buttons) is small compared to the benefits of receiving the service (100,000 hits in 24 hours), a market will exist for it one way or another, whether you're talking about drug-smuggling, prostitution, or selling Digg votes.

However, I think there's a way to fix it, and here it is. Have you ever seen people put a link in their profile to their HotOrNot picture, saying "Go here and vote me a 10!!"? Similar to the people who send links to their friends and say, "I just posted this, please Digg this for me!" The difference is that on HotOrNot, it doesn't work. On HotOrNot, you can cast votes for a picture in one of two ways. The first way is to go directly to the URL for someone's picture; the second way is to load the front page, where a random picture from the database is selected at random, and vote for whatever picture comes up. The catch is that the votes that you cast by going directly to someone's picture, are simply ignored in calculating the average score for that photo. The only votes that are counted are the votes cast for random pictures displayed on the front page. So if you want to manipulate the voting for your own photo, you'd have to load the front page hundreds of thousands of times waiting for your own picture to come up repeatedly, which is hard to do without being detected.

To enable an algorithm like this on Digg and reddit, the sites could present users with a sidebar box that displays random stories from the pool of recent submissions. (reddit already has a serendipity feature that users can use to select a random story from the available pool, which could be leveraged for this purpose.) Once a story has collected, say, 100 votes -- or whatever number is considered sufficient to provide a representative random sample of how the story appeals to people -- then on that basis the story can either be buried or promoted to the top, where it would be seen by, say, 100,000 people. The elegance of this system is that bad content would only be seen by 100 people on average before it's buried, whereas good content would be seen by all the 100,000 people who view it on the front page, so the average user sees 1,000 pieces of good content for every 1 piece of crap. Even if 75% of users ignore the random story box completely, that just means you have to display it to 400 users instead of 100 before you have enough data points for a good random sample.

I suggested essentially the same algorithm for how an open-source search engine could work without being vulnerable to gaming even by those who understood all of its inner workings. The main difference, of course, is that Digg and reddit actually exist now. Digg declined to comment on the possible merits of such an algorithm; reddit's Steve Huffman said that the idea sounded interesting, although even if the idea got full buy-in, naturally any proposed change would take a long time to bring to fruition.

But it seems that an algorithm similar to this one would be the only way to prevent cheating on sites like Digg that sort content based on user votes. So it's ironic that HotOrNot, the only site I know of that is using a variation of this algorithm and hence is probably the most secure against cheating, is also the one where cheating is least likely to be a problem. Getting a high placement on Digg might enable you to make some money, but getting a highly rated picture on HotOrNot isn't going to make you rich (unless it helps you meet a millionaire who is using the site to find his third wife). Also, making HotOrNot meritocratic doesn't give people an incentive to improve the "content" that they submit, because up to the limits of what can be done with hair and wardrobe, you can't make yourself that much more attractive. With Digg and reddit, on the other hand, I might work harder at submitting a good story, if I knew that it worked in a perfectly meritocratic fashion that pushed good stories right to the top.

If you do this, you don't need any of the other countermeasures listed in Annalee Newitz's follow-up piece "Herding the Mob", such as analyzing user account history for suspicious behavior. As long as most users in the system are legitimate, most of the users in your random sample will be legitimate as well, and their voting will be representative of what most of the community would think. A story could also get a high score within a specific sub-area of the site like the sports page, but kept off of the main site front page, if the story got a high score from a random sampling of sports-oriented users but a low score from a sample of everyone else.

You could even sub-divide the topical areas further, down to a level of granularity like "Would Barack Obama make a good president?" A site called Helium is currently trying something like this -- users can submit essays on subjects like "Racial inequality or oppression: Do they truly exist in todays society?", and vote on how to rank other essays against each other. The voting works on the random selection principle that I'm advocating here -- users are presented with a pair of randomly chosen essays from a given category (not necessarily the same category for which you submitted an essay) and told to vote for the better one, so there's no way to tell all your friends to go to the link for your essay and give it a high rating. The main limitation though is that while the votes can push you to the top of a particular sub-category, that won't cause your article to "break out" and get to the front page of the site -- Helium says that those front-page articles are chosen at random by employees from the among those articles that are highly rated within their narrow category, so just being good is not enough. And if you want to write something that doesn't fit into any existing categories, you have to create a new category for your essay like I did, which will then be a category containing one essay that nobody else ever sees. Perhaps both of these limitations could be overcome by adding the option to rate randomly selected essays on a scale of 1 to 10 -- thus providing a way to rate essays that exist alone in their own category, and also a way to find the best essays across the entire site, rated against each other.

If Digg or reddit adopts a model that uses the random-voter-selection method, then there's the issue of how to handle the votes cast by users under the current system -- the ones who go to a story link and click "digg it", which is what makes the existing system vulnerable to gaming. Digg could do what HotOrNot does, and just ignore those votes outright, but users would probably view this as deceptive. Perhaps Digg could say that votes cast by self-selected users (the ones who go straight to the story link) are counted along with votes from randomly-selected users, unless the average of the self-selected votes is significantly different from the average from the randomly-selected votes, in which case the self-selected votes are ignored. Hopefully this would satisfy most users and preserve the "community" feel of the site, and only a spoilsport would point out that counting the self-selected votes only if they agree with the randomly-selected votes, is exactly the same thing as ignoring the self-selected votes entirely.

I asked the owner of User/Submitter what he thought about this. He was willing to talk with surprising candor (except about things like his real name) and spoke as if he'd like nothing better than for Digg to make changes to their service that would block his system from working. To both Annalee Newitz and me, he said, "We find it interesting that Digg still allows anybody to view any user's diggs. By way of this 'feature,' User/Submitter is able to verify that our users actually digg the stories they're given. Without this feature, Digg users are given complete digging privacy, and User/Submitter cannot exist." Some have expressed skepticism that the Digg cheaters really want Digg to fix the problem. But as a security tester, I can understand that mentality. If you report a problem, and a company doesn't fix it, eventually you get tempted to publicize the problem to draw attention to it. And if they still don't fix it, and it's a fairly benign security hole that merely enables some pranksters to get some undeserved attention, why not build a service around exploiting the hole, if will highlight the problem and encourage it to get fixed?

So I'm going to go out on a limb and say the U/S guy sincerely wants Digg to be more secure. However I disagree with him about his proposed fix, that of hiding a user's digg history. First of all, it won't stop anyone who creates a multitude of accounts all under their control -- you can use Tor to make it appear that you're coming from many different IP addresses, and build up a history of "legitimate" votes before using your votes to push sites deliberately. (Be sure to use different browsers, or vary your User-Agent header if you know how to do that, so that a series of votes from identical browser types doesn't give you away.) If your service does work by paying other users to cast votes, then you could still audit whether they're casting their votes honestly -- for example, create a test story, use 5 sockpuppet accounts to digg it 5 times, then tell your confederate to digg it. If the number of diggs doesn't go up to 6, then you know they're not honoring their end of the deal, and kick them out of the system. As long as most confederates think there might be some chance of getting caught if they don't play along, most of them would probably cast the votes that they were paid for, since it costs them nothing to do so and they wouldn't want to jeopardize their stream of easy money.

I asked the owner of User/Submitter if his service could defeat the random-sampling algorithm I described. "It would slow down our service," he answered, "but certainly wouldn't eliminate it because eventually a U/S User will have an opportunity to vote on a U/S Submission by way of chance." But I don't see how this would beat the algorithm -- some U/S voters would still get to vote on the story, but as long as there are far more legitimate voters than U/S voters, then a random sampling will almost always contain far more legitimate voters. The U/S owner also said, "Randomized voting privileges would be unnecessarily confusing, frustrating, and fragmenting. Not to forget: unfair and undemocratic." Well, you could keep it from being "confusing" or "frustrating" by keeping the existing interface (with the possible addition of a randomly-selected-story box), so that the only changes would be in how the votes are handled under the hood. "Fragmenting"? If anything, it seems to me that the existing Digg/reddit algorithms would be more fragmenting, keeping users within their existing communities of friend who vote for each others' stories; a random-selection box would give stories with "crossover appeal" a greater chance of success, bringing them to the attention of users who might otherwise never have seen them. As for "unfair and undemocratic", presumably this is a reaction to the fact that the votes of 100 users decide what everyone else sees. But it's already the case with Digg that the votes of a small number of users decide what content becomes popular. At least with a random sample of users, it would be the case that the vast majority of the time, the voting outcome would be the same as it would have been if the entire site had voted, due to the magic of representative sampling.

So, I'm putting this suggestion out there for the same reason that Jim Messenger bought out Spike The Vote -- because I don't want sites like Digg and reddit to be manipulated by the abusers. In fact, if they used this algorithm, they would become more meritocratic than they are now, because the systems would strictly favor the highest-rated content, instead of content written by people who have informal networks of friends who can all go digg their stories for them. If I were to design the user rating system to make it cheat-proof, these are the exact details of what I would do:

• Wherever they decide to post the "random story sampling" box (on the front page, or on a link off to a separate page, etc.), have it work so that as soon as new stories are submitted, they can be rotated into that box and displayed to a random set of users, until it's reached its total of 100 votes or however many are required to get a random sample.
• You can have "shutout voting" to kill off stories early that are obvious spam or otherwise really useless, without going through the full 100 votes. (For example, if 90% of the first 10 votes are negative, then stop collecting votes.) This decreases the number of users "inconvenienced" by really obvious spam and other garbage.
• For someone to submit content that gets rotated into that voting process, have them submit a Turing test (read numbers off of a graphic and type them in), or something similar. This prevents spammers from submitting spam content over and over just to have it viewed by those initial 10 voters. If they have to type in a number each time, it's not worth it.
• When users give votes to a story, give them the option to say why they voted the way that they did. (This is especially valuable if they're giving negative votes, then the submitter would know what to improve.) Personally I think the comments would be more valuable if each user can't see other users' comments, at the time they submit their own comments; this prevents the "me too" effect where everybody echoes the first two commenters. (When I ask for independent comments from people, and they almost all say the same thing without seeing each other's comments, that's when I know they have a point!)
• To prevent an attacker from having their own username hit the random-voting page over and over in hopes of voting up their own content, make sure that each user account is only allowed to vote on a given piece of content once (even if they found the content through the random-story page).
• Require a Turing test for new user signups. This would prevent an attacker from registering a huge number of accounts just to hit the random voting page with different users over and over, in hopes getting to vote on their own submitted content eventually.

Then after running this system for a while, look through some collected data to determine if the system could be more efficient. For example, do you really need a sample of 100 votes every time? Suppose you determine that in 99% of cases, you get the same result just from tabulating the first 50 votes, as you would have gotten from tabulating all 100 votes. Then you could modify the system to collect only the first 50 votes, and then make a decision.

Suggestions for improvement? Flaws (hopefully not fatal)? Everyone who cares about keeping community sites like Digg free from abuse, and who wants to create a path for the best content to rise to the top, let's put our heads together and see what we can think of. The above is intended merely as a jumping-off point, and although I've worked it over and I can't see any specific points to improve efficiency, that's probably just because I've been looking at it too long. And if you Digg this story for me I'll give you 1,000 times as much cash as I gave my Mom last Mother's Day.

There's a cheaper way

This is what I like about slashdot. I can get my stories on the front page just by doing sexual favours for the administrators. No need to spend my hard earned geek money.

Re:There's a cheaper way

"Ya, all I had to do was suck it out of a hose."

Digg Down

Is there any way to digg this article down?

Re:Digg Down

Is there any way to digg this article down?

Yes: make sure that it's clear, concise, factually correct and that the masses don't agree with what you've written.

tl;dr

I think a better system would promote discussion over the actual articles, and perhaps form a culture in which reading the article itself is considered unnecessary. This negates most of the benefits of site-whoring (ad revenue). Maybe even build some sort of community moderation system to control the flow of discussion and keep interesting, insightful, and occasionally humorous posts at the top while limiting out posts of poorer quality.

Re:tl;dr

I don't think that would ever work. I mean... maybe, let's say if you somehow marketed it towards obsessive geeks who'd actually waste their time all day writing comments, moderating, and then what? You'd need a way to moderate the community moderation system, like some sort of meta-moderation. Who'd be a big enough loser to spend their time that way?

Not me. I'm not going to waste my time arguing about inconsequential topics on some dumb geeky community website.

Re:tl;dr

For starts this is a big "told you so" to Kevin from CmdrTaco... Kevin originally approached Taco about adding what would become Digg to Slashdot and Taco said it would never work... Both were wrong! Digg does work but would be much better if it took from Slashdot's experiences. This whole post is about the Digg/Slashdot p*** match. oh well, it gets page hits!

back OT, Digg's focus is to "Digg" around on the internet and find interesting stuff. Where Slashdot is about Quality, Digg is all about Quantity and cleverness. I find myself on Digg more (to follow the links than to discuss) because Slashdot has slowed down.. every time somebody post Lego robots, or weird news, etc (the "news for nerds" part of the slogan!) it gets flamed as "stupid" or not a "relevant" issue to the Slashdot (stuff that matters) "agenda". I've noticed from Firehose that clever quirky stuff just isn't making front page anymore even if it gets hits.

I'd say each site has it's place. In a lot of ways Digg "gaming" doesn't hurt if it doesn't happen too much as it's usually interesting stuff if somebody wants it there enough to go to all that work. On the other hand, Slashdot has better discussion. Slashdot has that cool factor of actual industry insiders that will show up and post... that's way cool and they only do it because Slashdot has that history. If Slashdot wants to remain "relevant" there's the ticket... get more actual people from the industry to post, answer questions, etc... that's where the NEWS really is.

How to Destroy the Competition, Slashdot Style

(1) Learn that one of your competitors can be easily manipulated to make its front page results 100% spam.

(2) As a "courtesy" to the community, post an article on Slashdot announcing this fact, describing the technique in detail, and offering a solution that will almost certainly be ignored by your competitor.

(3) Rub your hands in satisfaction as thousands more spammers are now made aware how easy it is to manipulate the competition.

(4) Watch the competition crash and burn as its signal to noise ratio plummets to zero.

(5) Profit!

terrible idea

I'm sad that someone spent this much time describing a system that seems like it would be a total failure. I think a system like this used on digg would actually give the users LESS influence in voting. Why? Because most users read digg from the standpoint of the most popular stories. They aren't interested in reading the dreck that hasn't been voted up yet. And believe me, there's a LOT of dreck. I certainly wouldn't waste time checking out the stories in the random box. The front page is random enough for me already. Something gets enough votes to be on the front page and then I check it out and digg it if it's good. But frequently crappy stuff will slide right off the front page if it's not really that good.

The problem is that the submitter is trying to translate a method that "diggs" a photo into a method that diggs a story. Story take much more time to read than photos. It simply doesn't work the same.

Used to be high on digg

I was really big on digg back in it's early days. It started as a small community of web devs and techy people. The early digg was dominated by good tutorial, cheap deals, and other stories of interest to web devs. Sadly it quickly went down hill once it became popular. It got overun with myspace kids and weird political conspiracy theorists. By about Sept 2005 Digg was dead. All my friends had left and there are only soo many videos of some dude getting hit in the groin, and so many "Bush is a moron who happens to be an evil genius" that intelligent people can take. Digg is the wasteland of the internet a collection of myspace level users posting stupid content. Intelligent users fled digg over a year ago.

Digg is lame anyway.

Digg is pathetic. The concept is democracy gone crazy, like those idiotic TV shows where the audience votes for who's the best performer. Whatever slashdot's shortcomings in other areas, at least they have paid editors who work at it like a real job. Reading the typical comments on digg, they all seem to be by high school students who think they know the secret to tabletop nuclear fusion, and they're all voting each other's posts up and down like crazy, based on nothing but their own biases. At least with slashdot moderation, posts are likely to be moderated by randomly chosen people who didn't get handed a license to go around voting their friends up and their enemies down.

Re:Digg is lame anyway.

Digg use to be pritty good. Then it really got bad.

Now when ever I check Digg. I normally get the following topic.
Look at the Cool Picture [cool, beautiful...]
The Bush Administration is Evil because...
Atheist are more moral people then anyone else.
Religious people suck because...

The Digg Systems of Moderation of comments is far worse then slashdots. At least in Slashdot most opposing viewpoints and not normally moderated down to base levels and usually spur debate. Digg if you defend Religion, the Bush Administration... You are quite quickly buried, and with little debate on the topic.

Before it was so popular Digg actually had some good and more intelligent stuff but not because it go so popular it is more average joe interest story.

Not quite

(Or, only sign up with manipulation services which publish a privacy policy promising never to share your information, especially not with sites like Digg. Then if Digg buys them out, then the site has violated their privacy policy and Digg as the new owner inherits the liability for that, so you can sue them, right?)

That's a nice piece of logic to prevent your information from being shared, but I don't think it would work that way. If Digg bought out that service, then Digg becomes the new owner at the same moment that it becomes privy to the information; and a company cannot be held liable for sharing information with itself in the absence of a no-data-retention agreement. Even so, how many companies out there do not have a we-reserve-the-right-to-change-this-agreement-with out-warning clause attached to their privacy policies?

Unfair comparisons

I know there are those that compare Slashdot and Digg and they couldn't be more apples and oranges. I come to Slashdot when I would like some intellectual insight and I go to Fark when I want something funny.

I go to Digg when I want OMG!!!! L33T Fanboi yatayata...
Digg is, in short, a wasteland. The submissions are great but if one wrote the book on it, "Pooh foraging for honey" would read like Tolstoy.

StumbleUpon beat you to it.

There's actually a website that does exactly this already. It's called StumbleUpon [stumbleupon.com]. You click a button and you're brought to a random page according to one of your many subscribed interests. If you like it, you say so, if you don't you vote it down. Bad pages don't get any up votes, or may even get down votes, and so they are quickly weeded out. Nice pages get some number of up votes and then subsequently get shown to more people who in turn might give them more up votes and so on.

Re:A good design

Or, to combat the moderation services, just sell the top two positions (ala google) and be done with it.
People can preference it out as it could have the topic :paid: or dis-allow that and risk alienating part of your user base.
revenue and it neuters the outside manipulators.
-nB

Re:A good design

Roland? Is that you?

Re:A good design

If by that you mean 'enforce the groupthink' then you are correct.

Slashdot's readership tends to mod up posts they agree with and mod down those they don't, regardless of whether the posts actually further the discussion on the topic. Of course the editors can manipulate that to ensure that the site stays on topic and appeals to the target demographic. I am not claiming that this is a bad thing but the system certainly doesn't just run itself, nor does it generate any real discussion of anything.

I am probably less emotionally invested in my online personality that most people but Slashdot has plenty of users who will respond to their own posts complaining about some moderation they have received. I don't see why it matters to anyone. A person could create a new account and quickly build excellent karma by recycling highly rated old comments from previous stories, never even having to think up a post on their own. There are more than enough moderators who just want to see things they agree with rise to the top regardless of whether they are getting their blocks trolled off.

That, lame jokes, and endless learning by metaphor is really what drives the site.

Re:A good design

It does happen, but I've found in my own experience that I'm often modded up for posts counter to the Slashthink.

I hold (or at least, express on Slashdot) contrarian views on several issues. I try to express them reasonably and politely, and find that I'm often moderated up for those posts, even though most of the other posts express the opposite views. It's as though there's a large but fairly silent majority on those issues, people with enough karma to have mod points.

Yeah, some days I get moderated down well under water before that happens (usually when I'm being too snide or to subtle in my sarcasm), but that's not often, and my karma can take it.

I can't guarantee that any particular posting will survive the whims of the moderators, but it seems to me that a whole view is rarely suppressed: somebody's similar postings will be modded up. (If you're the only one on Slashdot who thinks a particular way, then you're probably wrong about whatever it is. There's always somebody out there who agrees with you unless you're batshit crazy.)

I think that the proof of the pudding is in the eating: I find that although the Slashthink often infuriates me, there are enough insightful comments that it's worth going in to many articles. That's especially true when I feel there's something missing in the story, e.g. some energy breakthrough which I'm certain is overplayed but I don't know why.

Re:A good design

For as many problems as Slashdot may have, things are exponentially worse on sites like Digg. Digg is a great site in its own way, but for actual information I generally tend to come to Slashdot knowing that I can expect to find insightful comments and a knowledgeable community that generally has something to add to the discussion. While there are certain people who will moderate poorly, through the meta-moderation system there is the chance that this will be recognized and stopped.

There are also a number of other tools that Slashdot offers that can help you while browsing the comments. Don't like reading the Soviet Russia and Futurama reference jokes? Just change your preferences so that comments moderated as funny are weighted less heavily. If you find a user that seems to get moderated insightful for comments that you feel are trolls, add him as a foe and weigh such posts less heavily. The simple fact that there are options like this on Slashdot make it so much better than other sites when it comes to the quality of comments.

The only other than I can do is to encourage you to be a good moderator. Maybe there's something you agree with or find interesting, but it's already been moderated as such so it might be more worthwhile to sink your points somewhere else. The simple fact of the matter is that any system implemented, no matter how good on paper, is still at the mercy of people who don't always tend to act rightly. Slashdot will never be perfect, but I enjoy it for what it is, and that's why I continue to come here.

Re:A good design

That, lame jokes, and endless learning by metaphor is really what drives the site.

What's the problem, I never metaphor I didn't like! (nyuck nyuck.) But seriously, a community site without metaphors is a train wreck waiting to happen.

MOD PARENT UP

Parent is right, there is NO way to abuse the moderation system.

Re:A good design

Hey, I am a Karma whore you insensitive clod!

-Rick

Re:A good design

Since I don't get paid for it I guess I'm actually a karma slut.

Re:A good design

While this maybe true, digg's voting is also very poweful (and I say this as someone who has had a slashdot account for a long time and checks it far more than digg). As mentioned in the editorial, bogus or lame stories get buried pretty quickly.

Adios karma!

As an ex-Digg user, I noticed a totally different trend. These days the top stories are lame videos or pictures. Digg is no longer about news, it's about what people think are cool videos and pictures. Digg is worthless to find news. Not to mention the comment section, I lose 20 IQ every time I read something there. At least on Slashdot you can find interesting stories and some very insightful commentaries. Digg also has zillions of blog spams, people posting their worthless blog posts where they link another article and whatnot.

Digg has almost no content worth reading.