Bram Cohen's Response to Microsoft's Avalanche

An anonymous reader writes "Bram Cohen has reduced Microsoft's proposed file-sharing application--codenamed Avalanche--to vaporware, dubbing its paper on the subject as "complete garbage". "I'd like to clarify that Avalanche is vapourware," Cohen said. "It isn't a product which you can use or test with, it's a bunch of proposed algorithms. There isn't even a fleshed-out network protocol. The 'experiments' they've done are simulations.""

Why The Rant?

[CleverNickedName]

"It isn't a product which you can use or test with, it's a bunch of proposed algorithms. There isn't even a fleshed-out network protocol. The 'experiments' they've done are simulations."

Doesn't all software start off this way?

Avalanche

[Dink Paisy]

It's not even a code name. Avalanche is an academic research product coming out of Microsoft's research organization. I guess there are not plans at Microsoft to make Avalanche into a product. If it were ever released, there is a decent chance it would be shared source, since researchers tend to like that kind of thing.

But, yeah, like he said. Avalanche isn't supposed to take over the world. It isn't a product, and it doesn't exist in source code form.

Who?

[pr0nbot]

That's Bram Cohen, author of Bittorrent.

Re:Pointless Article

[spellraiser]

Another pointless article. Troll me, but the fact is that this is addressing something that is behind MS's closed doors.

Ah, you mean like this research paper [microsoft.com] that Cohen is criticizing.

Or perhaps you are referring to these completely unfounded claims (from TFA):

The developer said Microsoft had completely misunderstood the way BitTorrent operated. The paper quotes "the tit-for-tat approach used in the BitTorrent network" as an inspiration for parts of Avalanche's own operation. Under the approach, a peer-to-peer client will not upload any content to another client unless it has also received a certain amount of content in return.

Cohen said, however, this was a waste of time and had been discarded long ago.

"I can't fathom how they came up with this," he wrote. "Researching either the source code or the documentation on the BitTorrent Web site would have shown that the real choking algorithms work nothing like this."

"Either they just heard 'tit-for-tat' and just made this up, or they for some odd reason dredged up BitTorrent 1.0 and read the source of that." BitTorrent is currently at version 4.0.2.

Cohen went on to say that the 'tit-for-tat' approach was used when BitTorrent was still being developed, but that the first real-world test with only six connected machines showed that it did not work well.

Yup, that's a guy bashing closed doors alright.

Has anyone even read the MS paper?

[Anonymous Coward]

I just took a look at the MS paper and they propose using "network coding" to eliminate block scarcity. The idea is that each node transmits out a linear combination of the blocks they've downloaded, along with the coefficients used. When enough of these are gathered, the system can be solved and the file reproduced. This actually seems useful and is not "complete garbage".

fwqcwq

[rbarreira]

Yeah, I know most of the posts here are bashing Cohen for bashing microsoft (and I was agreeing with them until I decided to RTFA - the summary is not a good one). Bram isn't randomly attacking them for having a vaporware product, he is specifically pointing out the many mistakes that they make in their paper, where they compare Bittorrent to their proposed algorithms. It seems that they made too many mistakes to make their research paper valid, so their simulations are crap... RTFA!

Re:Why The Rant?

[Jerf]

Skip the article, read the blog entry [livejournal.com]. First line:

A bunch of people have been pestering me about Avalanche recently, so I'll comment on it.

A perfectly reasonable reason to discuss something. It isn't Bram that posted this to Slashdot.

I think he's trying to point out to the "bunch of people" that at the moment, Microsoft isn't exactly shipping the BitTorrent killer that he's somehow "got" to respond to. He might get less dismissive if they ship something that obviously works.... or if people didn't pester him.

(I've seen several people comment that Bram's "arrogant"; it's nothing to the arrogance of assuming they can force him to comment on something, or the arrogance of assuming that his essay was written straight for them, or the arrogance of saying since they don't like it it shouldn't have been written. This is just an addenda so I don't have to post again, not directed at CleverNickedName.)

Summary

[The New Andy]

1) The MS paper used the wrong model for the choking algorithm (it assumed tit for tat). This method isn't used because it sucks (as discovered early in the life of BitTorrent.

2) The paper also assumed that each client would only try to connect to 4 peers. Bram says that 30-50 is more realistic.

3) In spite of the poor comparison, the ideas might be useful.

The actual blog entry

Media distribution. That's why

[CdBee]

Windows Media has been prepared for several years as a leading format for use in pay-to-view downloaded material. Microsoft even developed Windows media centre to run TV-connected PCs.

What's missing is the distribution technology. Even with modern 8mbit DSL / Cable connections, an HTTP or FTP download of a 900mb movie file is very expensive for the company hosting the software and files. However, if each set-top-box or WMC PC has a secure file-sharing system preinstalled, then most of the upload bandwidth can be shared among people who have already downladed the same file.

Consumers will hate it - especially as upload bandwidth is often slow and overall bandwidth capped - but the media distributors will love it to bits.

It's not MS bashing!

[Limburgher]

Bram read a paper on a topic he actually knows something about, and critiqued it. It's not like he did this sight unseen and is just venting sour grapes because the big bad corporation is trying to steal his thunder. He actually judged MS's proposed software on the available merits, and found it deeply wanting. He also goes to point out a few minor strengths. Yes, all in all he relegates Avalanche to the junk heap, but he did so based on a surprisingly dispassionate evaluation.

So, lay off! :)

Re:Respect in the industry

[Anonymous Coward]

RTFA

First sentence:
`` A bunch of people have been pestering me about Avalanche recently, so I'll comment on it.''

So much for that.

Re:Who cares that it's vaporware?

[Alioth]

No, it's not good vapourware according to Bram Cohen. He picks several large holes in the white paper.

Re:SDLC

[Stauf]

Did you read the response you refer to [livejournal.com]? Avalanche claimed to be superior to BitTorrent and based its argument on assumptions and old code. Cohen corrected the misconceptions.

He also went out of his way to explain why Avalanche is doing things wrong and where their testing methodology had come up wanting.

Re:It's NOT vapourware

[-brazil-]

I think you're overstating the case. According to the jargon file [retrologic.com], vaporware is mere " Products announced far in advance of any release (which may or may not actually take place)", i.e. malicious intent, or even lack of intent to release, is not necessarily implied. Wikipedia [wikipedia.org] elaborates that apart from the cases you describe, it can also be a "test ballon", with the project getting cancelled when there is not enough positive response, or simply the result of too much optimism.

Have you even read Cohen's article?

[Anonymous Coward]

I just took a look at Cohen's article and he responds exactly to that proposal and elaborates why it does no good and can actually be bad.

RTFA

Re:And there it is!

[MorningDew76]

Try using a legitamite source for BT.. such as bt.etree.org (and i don't care if you think that music sucks)

you will find that the popular torrents on there FLY, and that's because of the mentality of the userbase. they like to share, and don't hop off a torrent right when it's done. there is nothing illegal about it, thus no fear of the man knocking down your door. i've left seeds on there for months.

so.. if you are getting crappy speeds, i'd recommend finding yourself a better group of people to swarm your files with. bashing on BT isn't going to solve your problems, and niether is a piece of vaporware from microsoft

Vaporware: Par for the course for most R&D lab

[steve_l]

I work at a corporate F100 R&D lab, most of what gets written up is either

-prescriptive stuff that you'd like adopted
-things you built that you'd like the world to know about.

For a corporate group, a paper is only a half-success, depending on the ranking mechanism. A Popular paper is good, but not as good as getting into shipping product. And there MSR have the same problem I have -the gulf between research code and production stuff. Actually, their problem is worse, they have to go through the MS lifecycle, whereas our codebase is now open source (http://smartfrog.org/ [smartfrog.org]) so that we can have stuff in users hands in real time.

Summary: the presence of an MSR paper on its own is meaningless.

returning to MS and P2P, note that MS own groove, which has an excellent P2P filesystem, though one that will forever be windows only. They already do P2P products, they just are not as common as say, Exchange Server.

Re:And there it is!

[pocopoco]

This choking algorithm may be beneficial in the sense of increasing the total connections that a seed will accept but it robs the system of its performance benefits.

Seeds do not use choking. Choking is used by peers without the complete file on peers that aren't sending them data. Seeds need no data and so do not perform chokes. Last I looked (admittedly an early version) seeds will send to the clients that dl the fastest and will only send to a small number of clients at a time for efficiency reasons.

Super-seeds are completely different (but still don't use choking, although they reward people who received a piece that the super-seed detects has been spread around well by the people who received it).

I can't believe you typed a whole rant about choking without having the slightest clue how it is used, however. You could have spent that time googling and a) learned something and b) not come across as an idiot.

Re:bad research, too

[nonlnear]

Actually, Avalanche is a FEC method. The advantage of Avalanche over other FEC methods is that the server doesn't have to do all the coding. Hence the term `network coding'.

While the paper didn't worry too much about comparing Avalanche to other FEC methods, the comparison seems moot, as server coded FEC methods seem obviously impractical for individuals wanting to seed data from a humble PC. Reliving the seeder of the burden of coding seems an obvious enough differecne that Avalanche and other FEC methods are not nearly as apples to apples as comparing to Bittorrent. (Because Bittorrent is actually practical for Joe Celeron-user to seed right from home.)

Implementation may end up being harder, as it will be a lot harder to combat poisoned blocks in Avalanche. I think the authors were too optimistic about this issue.

Re:Remember...Bram is Autistic

[rbarreira]

Are you sure Theo isn't?

I'm also not so sure that those comments are related to his disorder...

(btw Asperger's syndrome [wikipedia.org] is a very mild form of autism in case someone was amazed by reading the parent post)

Re:bad research, too

[PureFiction]

They did attempt demonstrate how this is superior, by distributing the error coding to the clients (network coding) so that novel bits are rapidly incorporated into subsequent blocks, and also a secret method (read: patented) to authenticate distinct blocks generated by clients, which is pretty tricky. this is a vast improvement over centrally generated FEC blocks with authenticated checksums, which is not an improvement over bittorrent.

Re:Avalanche

[ad0gg]

Microsoft never released a press release on avalanche. Hell its not even on research's main website. Its on some guys personal directory. [microsoft.com]

Re:Not even close to finished, you say?

[locofungus]

Using a (254,127) RS code over GF(2^8) it is possible to recover all 254 symbols from any 127 where each symbol would be a byte.

All 254 symbols are linearly independent

This would then be run in parallel for all bytes in a block.

Re:Newsflash

[grcumb]

"Quotes like "The lack of any concrete numbers at all shows the typical academic hand-wavy 'our asymptotic is good, we don't need to worry about reality' approach" certainly don't earn him much respect from academics in system programming research who work very hard, thankyou very much, to ensure that their results are realistic. He has turned a simple observation about the paper (they neglected certain overheads) into a bigoted rant (academics are foolish). Not cool."

I'd venture to suggest that he's tired of 'typical academic hand-waving', in which 'neglected overheads' get overlooked. Take this nugget, for example:

"The really big unfixable problem with error correction is that peers can't verify data with a secure hash before they pass it on to other peers. As a result, it's quite straightforward for a malicious peer to poison an entire swarm just by uploading a little bit of data. The Avalanche paper conveniently doesn't mention that problem."

The paper exposes a fundamental misunderstanding of the process it's attempting to explain, doesn't have the adequate means to test its assumptions - and still should be taken seriously by the one person who has the most experience with the kind of problem it purports to 'solve'?

No, I think it's a simple matter of Bram being right [goofalicious.com] and refusing to suffer fools gladly. He does not generalise illiberally; he provides reasons for his disdain, and makes it clear it's based on experience, not assumption:

"I'd comment on academic papers more, but generally they're so bad that evaluating them does little more than go over epistemological problems with their methodology, and is honestly a waste of time."

So kindly quit with the ad hominem attacks. Bram's attitude may be dismissive. It's his right, and it does nothing to reduce the impact of his observations, which expose just how badly the researchers have misunderstood the issue.

Bram is one of the giants on whose shoulders we have the privilege to stand.