Swarmcast GPLed

miguel writes "OpenCola has just released SwarmCast which is a very interesting mechanism for distributing software. At Ximian we are looking into integrating this into Red Carpet to accelerate software downloads by using their sharing software. The demo of their product is pretty amazing." Very clever: essentially it creates a peered network so larger files can be shuttled around faster. Each client can serve a small piece of data to other clients so that a massive centralized data center isn't necessary. Now the cola on the other hand...

Re:Ok, but...

Swarmcast is neither a fragile chain structure, nor a hierarchy, it is a many-to-many 'swarm' structure where peers send and recieve data from many peers in parallel. The use of Forward Error Correction allows us to have a potentially huge number of unique packets in the mesh where only a small subset of those packets are needed to recreate the original content. This allows the peers to swap data back and forth in a fairly random fashion to provide a high level of resiliance against changing network conditions, very high throughput, and rapid scalability.

-Justin Chapweske, Lead Swarmcast Developer

Re:What's with this gateway thing?

Everything you need to both serve and download content is released under the GPL. Besides, its peer-to-peer so there really isn't that much of a "server" concept. The gateway is mostly for content management and permissions, the kind of stuff that companies pay money for so that I can keep my job and write more open source code. -Justin Chapweske, Lead Swarmcast Developer

Gott in Himmel!

Man, you don't usually see download rates like that from anything except akamai. Problem was, the data got corrupted.

And no, it's not like a bucket brigade. It's more like building a house with more than one bricklayer.

Re: slow hosts-- if I understand correctly, all swarmcast hosts maximize their available bandwidth, automatically balancing the download.

Each node downloads packets one at a time. If a host is slow, it won't grab as many packets, and someone else will.

If machines A and B are downloading, and A is 5x as fast as B, then in the same time, A will download 5 packets and B will download 1. If there are 12 packets in the file, A will download 10, and B will download 2.

Re:Possible Security Issues

I don't think any of these are real issues:

First of all, what type of security is going to be implemented to prevent hacks. It seems that it would be pretty easy to shore up a single server, or even several in a single datacenter, but it would be a daunting task to protect thousands of machines spread thoughout the world against hackers.

So long as the swarmcast client software doesn't have any holes, how is this an issue? If it does have holes, yes, this is a big deal, but if you've got a machine on the net with other security holes, that's an entirely different problem.

Secondly, what type of redundancy is going to be built into this system. Again, if a file is going to be served from a single centralised machine, it should ideally be fairly reliable, with multiple connections, and RAID to ensure continuous uptime. However, if you're serving tiny pieces of a file from thousands of boxen, it seems to me that if even one of those fragments doesn't make it all the way downstream, the whole file would be worthless. Has Swarmcast done anything to prevent this from happening.

This is a pretty trivial problem to solve. There's still got to be a central server somewhere that tells you where to look for the peer bits. I don't know the details of their setup, but it certainly seems trivial to implement MD5-checking per chunk, so you can tell if there are bad chunks in your download. You just have to trust the main server's md5 signatures, and never end up trusting the peer servers.

This applies to the first point, too. If a serving machine is hacked, it's not a problem beyond that box because the real security comes from the MD5 key, or equivalent. It may be possible for a hacked server to serve bad code, but, given reasonable client design, it should be impossible for this bad data to actually be used by another machine. (Or, at least, should be impossible without big flashing lights and sirens screaming "You fool, don't do that!" :)

Re:Possible Security Issues

Erm, I'll bite.

Secondly, what type of redundancy is going to be built into this system. ...if you're serving tiny pieces of a file from thousands of boxen, it seems to me that if even one of those fragments doesn't make it all the way downstream, the whole file would be worthless. Has Swarmcast done anything to prevent this from happening.

One of the main benefits of distributed anything is that there's an incredible amount of redundancy because every node is willing to contribute a little towards achieving your goal.

Are you trolling?

First of all, what type of security is going to be implemented to prevent hacks.

Against what? To prevent against file modification, checksums can be distributed from a single point since they're much smaller. All other security problems will be the same as any other P2P system.
--

Freenet solves security concerns

Why not distribute the pieces on Freenet? Using SSKs you can be sure that each piece was inserted by the original poster. Maybe this would make a cool client for Freenet, though...

What's with this gateway thing?

What's with the "swarmcast gateway" that you have to use? If all we're getting is a GPLed client to an ultra-proprietary [swarmcast.com] secret server, forgive me for not getting too exited.

Still a cool idea, but if OpenCola wants everybody to put eggs in their basket, maybe somebody should release a GPLed server too...
--

Re:First Impressions

Well, my first problem is that the FTP server for downloading the JWS file seems to be /.'ed, so I can't get that and can't install SwarmCast, which would have solved the problem in the first place. Hmph...ironic, in a Catch-22 kinda way, isn't it? ;)

nlh

License

Sure, their client may be under the GPL but some of the terms on their license agreement are quite unpleasant:


(a) Reverse assemble, reverse compile, or reverse engineer the Gateway (or any component or portion thereof), or otherwise attempt to discover any underlying Proprietary Information (defined below) of the Gateway;


Isn't reverse engineering explicitly granted by law?


(b) Sublicense, rent, sell, lease or otherwise transfer the Gateway (or any portion thereof) to any third party;

(c) Remove or alter any marks or designations indicating the ownership of copyrights, trademarks or other intellectual property rights of any party contained in the Gateway;


In fact I'd go so far as to say these conditions wouldn't feel out of place in a Microsoft EULA.

Could this be used as a "legitimate use of p2p"?

The EFF is seeking help in this area of finding legitimate uses of peer to peer technologies...

Maybe this might be a very good argument in court... content distribution at high speeds.

Saving content providors money?

No thanks, if a content providor wants to provide files, they can pay for the bandwidth. If I want to download that file, that's my choice and thus I pay for the bandwidth it takes to download it at my end. However, just becasue I decide to download the file doesn't automatically mean I want to serve this file to anyone and everyone else.

This saves the content providor bandwidth at my expense. Bandwidth is, of course, not free and somebody has to pay for it. Ximian take note, if you don't want to pay for the bandwidth involved in being in the software distribution business, get out of that business. For Joe Average that installs Red Carpet and doesn't understand that it now includes this new feature, is going to be mighty pissed when he gets his bandwidth bill only to discover that he's been serving 10 gigabytes to strangers due to this feature.

Possible Security Issues

Maybe it's just me, but the concept of a distributed file server just doesn't sound like a very good idea. I have several reservations about this software to which I couldn't find any acceptable answers on Swarmcast's site.

First of all, what type of security is going to be implemented to prevent hacks. It seems that it would be pretty easy to shore up a single server, or even several in a single datacenter, but it would be a daunting task to protect thousands of machines spread thoughout the world against hackers.

Secondly, what type of redundancy is going to be built into this system. Again, if a file is going to be served from a single centralised machine, it should ideally be fairly reliable, with multiple connections, and RAID to ensure continuous uptime. However, if you're serving tiny pieces of a file from thousands of boxen, it seems to me that if even one of those fragments doesn't make it all the way downstream, the whole file would be worthless. Has Swarmcast done anything to prevent this from happening.

Hello, thats the wrong license

You've successfully quoted the Sun Java Web Start license. Now please click the word 'license' in Swarmcast's about box and you will see that there are no such reservations or claims.

You are the weakest link. Goodbye!

First Impressions

Well, I just finished downloading their sample file (a 20MB quicktime (uugh) trailer), and the technology seems to work, albeit not as well as a single fast server. When the download started there were 3 participants in the 'active mesh' and I was getting 41KB/sec or so transfer. By the time it was done there were 12 participants, and I finished the download at 55KB/sec. Even after the download finished, the Swarmcast client is sitting in my tray (Yes, this is a windows box) happily transmitting at between 8 and 14KB/sec.

Thing is, I don't see why anyone would use this. In reality, the transfer rates aren't as good as a single fast dedicated server (I can easily get 75-80KB/sec on this line), so there's really no gain on the client end from using this. On the server end, yeah you're using a lot less bandwidth. This might be useful for open source projects or other products that are downloaded by knowledgable people, but you're average computer luser isn't going to want to download 6MB of Java Runtime/Swarmcast Client just so they can save your company money by getting slower-than-normal downloads. (And I shudder to think what a distributed p2p network comprised mostly of 56k modems would be like, at least at the moment it seems most of the users online are broadband.)

Anyways, pessimism aside, here's what I'd like to see:

• Some indication of how much bandwidth you're 'serving'. (My estimate above is based on the stats in my RRAS console, there's no indicator anywhere that you're even sending data.)
• An option to automatically shut the client down when you're done with the file. This may seem selfihs, but I still know people paying by the hour for 'net access, and until they can be sure they can start a download with this and have their connection go idle when it's done, they're not going to use it.
• Some sort of statistics (Number of packets received, number served, total clients served, things like that.)

OK, I'm done for the moment. If they play this right I can see it helping out smaller outfits with knowledgable users, maybe eventually even going mainstream if they can convince people it's worth their while to install. It's definitely sparse on information in it's current form, but hey, it's a beta. It's a good idea, and I wish them luck.

-Jade E.