PDTP - The Best of Both FTP and BitTorrent?

ikewillis writes "For awhile I've been following the development of PDTP (Peer Distributed Transfer Protocol), which is trying to merge the concepts of FTP and BitTorrent. This sounds like it could be useful for apt-get repositories or other high demand FTP sites. It's designed to be used as part of scalable networks which could replace manual selection of FTP mirrors. It also supports a number of other nifty features like cryptographic file signatures. Isn't it about time we ditched FTP for something better?"

from the no-more-april-fools dept.

Too late! Tin Foil hat firmly in place!

Next thing it'll be transmitting voice and pictures over radio waves... AS IF!

Re:from the no-more-april-fools dept.

Tin Foil hat firmly in place!

How does this differ from any other day of the year?

This isn't fair...

I feel sorry for these people. See, this isn't your typical slashdotting... It's a slashdotting that comes after eighteen consecutive nonsense stories being posted over twelve hours on the US April Fool's Day.

So, their chance to build a reputation is going to be damaged by the fact that anybody reading Slashdot today has already given up on finding anything useful, and will be evaluating them as a joke that they're "not getting" rather than as a proposed networking scheme.

Furthermore, the geek world is bored today by Slashdot's denial-of-normal-service throughout the day. So, once word leaks out that this is a real and normal story, they're going to get all of the pent up slashdotting force applied to their server.

Simon, you should have started your set tonight with an NY Times article or two. That would have been a suitable transition between nonsense content and factual content, since NYT operates in that murky space and has a suitable web setup to absorb a larger-than-usual slashdotting. I'm sure the people at PDTP would have not minded at all if their moment in the sun had come an hour later tonight.

Re:This isn't fair...

Hey Cluster, you invented a new term that requires canonization. "Denial-of-normal".

Submit it to wikipedia. http://en.wikipedia.org/wiki/Internet_slang

Re:This isn't fair...

Perpendicular?

SB

Re:This isn't fair...

No way Dude!!1 Anyone can see that they have been working on this April fool's joke since last November!!

Re:This isn't fair...

You people sound like a senior citizen mad about loud fireworks on the fourth of july, or someone pissed that a Christmas walk has main street closed off.

Do you know how much Andy Rooney gets paid?

Re:This isn't fair...

No kidding, today has been ridiculous.

There's a point where it's funny, but then there's a point where it's really just overdone. I haven't even read half the stories posted, but it seems like they're all fake... and if they aren't, Slashdot is really ruining the credibility of some people. Not sure if the BSD on Gameboy is real or not, but if it is, who's gonna believe it?

You don't see CNN taking the day off. "OSAMA BIN LADEN CAPTURED!!", you click the link and read a long drawn out story that COULD be true, but at the bottom: "...April Fools!"

Karma to burn. *shrugs*

Re:This isn't fair...

There's a point where it's funny, but then there's a point where it's really just overdone.

Hang on...

Don't they overdo it every year and isn't that part of the joke? Maybe not an April Fool's Joke but rather an April Troll? (Which is obviously more appropriate for a geek site)

That's what I always assumed.

Cheers
Stor

BitTorrent Mod That's Neophyte Friendly

BannedMusic.org [bannedmusic.org] made a BitTorrent wrapper that installs the application and then automatically launches the download, they call it an "easy downloader" and have instructions and a script [bannedmusic.org] for sites that want to make their own. Makes it a *lot* easier for sites to give out big files to non-techy audiences.

need lightweight clients, not installers

BannedMusic.org made a BitTorrent wrapper that installs the application and then automatically launches the download, they call it an "easy downloader" and have instructions and a script for sites that want to make their own.

And unfortunately, it's windows only, and still requires installing the software, which is 3MB+.

What is needed is something along the lines of a very small, very simple java client or a browser plugin. Azureus is java, but is huge and has massive feature-bloat for the purposes of just downloading(and sharing back) one file. However, Bram and others don't seem terribly interested in expanding possibilities; a mac developer offered up numerous improvements to the BitTorrent team for the mac client(which among other things is based on 3.3a, not 3.4.1, weeks after 3.4.1 released) and was rewarded with deafening silence.

The bittorrent protocol is http based. It's extensively documented on the bitconjurer website. Cmon folks, let's at least see a mozilla plugin or something! :-)

You forgot one line:

I look forward to the community's response.

apt-get, you say?

Quoth the Debian Troll [slashdot.org]:

Even though the apt-get code is GPL'ed and therefore available for all to read, the majority of people miss a few subtle points in the source code. The assembly optimizations. The OpenGL hooks. The MP3 streaming capabilities. Instant messaging interfaces. Links to satellite tracking networks. I believe apt-get to be on par with such great open source works as the Linux kernel, Apache, and xbill. It is about the package format. It is about apt-get. It's about standing up and saying "Dammit, I'm sick of RPM not having any cluster management capabilities or Mac OS X Expose-like animations, I'm mad as hell AND I'M NOT TAKING IT ANY MORE!"

raid

R.A.I.D. == redundant array of intolerable diversions
or at least on april fool's day....

PDTP

PDTP eh? Try saying that too quickly.... I can barely get my tongue around half these transfer protocol names.
I wish people would mind their T's and P's.

Re:PDTP

"I am Bittorrentholio...I need PDTP for my Torrenthole!"

"Heh heh heh heh...file transfers RULE!"

The concept is great, but...

...mirrors would need to be in sync at all times for this to work. Otherwise your PDTP client is only able to download from the mirrors that are in sync, or worse, will get some chunks from files that aren't up-to-date, causing problems.

Unfortunately, it's (almost) impossible to mirror new files instantaneously, so mirrors are never all in sync, all the time.

Re:The concept is great, but...

I'm assuming that you're going to raise that potential problem with the pdtp developers, right?

Sorry, pet peeve is people kvetching about something on /. but not telling the developers.

To fix this, perhaps they could mandate that mirrors copy a particular directory to a temporary location, then take the old directory offline for the few minutes it would take to copy the new files over. Or have a $RELEASE var that clients would ask for and get returned all files marked with that var.

or something. ;-)

Re:The concept is great, but...

Hello. I'm the designer of PDTP.

...mirrors would need to be in sync at all times for this to work. Otherwise your PDTP client is only able to download from the mirrors that are in sync, or worse, will get some chunks from files that aren't up-to-date, causing problems. Unfortunately, it's (almost) impossible to mirror new files instantaneously, so mirrors are never all in sync, all the time.

I suggest you look at this page with graphic illustrations of PDTP networks [pdtp.org] for a better idea of how PDTP works. There is no concept of a "mirror" in a PDTP network. The Source Server is the central authority on all files being distributed over the network, and notifies all servers/piece proxies on the network whenever files become available or unavailable. Like BitTorrent, the network is largely self-sustaining, with clients uploading pieces to each other and verifying their integrity with MD5 or SHA1 checksums. Files are tracked on the network with integer keys, so if a file were altered its name would simply be mapped to a new key, and the entire network would be notified that the previous version is no longer available.

Hash trees!

Please don't use straight SHA1 - it requires downloading the entire file to verify.

Bittorrent and some other file sharing networks split the file into chunks and keep metadata with the hashes of chunks. The problem with this idea is how big to make the chunks: too big and you need to download a big chunk before you can verify. Too small and the list of hashes itself takes too long to download (the hashes are what makes .torrent files relatively big).

I think the solution should be to use hash trees. Split the file into relatively small chunks (1k?) and calculate their hashes. Now take every two consecutive hashes and hash them. Repeat with the hash results from the previous step until you have a tree with a single hash at its root. The root hash represents the entire file just like an MD5 of SHA1 sum. The difference is that with a small amount of metadata as hints you can verify any part of the file without downloading the entire file. All you need is a short (log n) chain of hashes leading down to the root hash. The server will trickle the hash information interleaved with the download and the client will verify it on the fly and never need to write a single byte to the disk before it's cryptographically verified.

think about that sentence:

Isn't it about time we ditched FTP for something better?

Isn't it about time we ditched floppy disks for something better?

Isn't it about time we ditched IDE drives for something better?

Isn't it about time we ditched x86 for something better?

Isn't it about time we ditched Microsoft Windows for something better?

Isn't it about time we ditched CDs for something better?

Isn't it about time we ditched telnet for something better?

Isn't it about time we ditched CRTs for something better?

Isn't it about time we ditched 20-year-old TV sets for something better?

Isn't it about time we ditched COBOL for something better?

Isn't it about time we ditched BASIC for something better?

Isn't it about time we ditched SCO Unix for something better?

Isn't it about time we ditched DOS for something better?

Isn't it about time we ditched Dubya for something better?

my point is that there is a lot of very old crap out there that should be replaces, but is going to get used and keep getting used for years to come.

Re:think about that sentence:

Isn't it about time we ditched floppy disks for something better?
CD-RW

Isn't it about time we ditched IDE drives for something better?
SATA

Isn't it about time we ditched x86 for something better?
AMD

Isn't it about time we ditched Microsoft Windows for something better?
Linux

Isn't it about time we ditched CDs for something better?
DVDs

Isn't it about time we ditched telnet for something better?
SSH

Isn't it about time we ditched CRTs for something better?
LCDs

Isn't it about time we ditched 20-year-old TV sets for something better?
New TVs, available at your local stores.

Isn't it about time we ditched COBOL for something better?
Visual Basic.

Isn't it about time we ditched BASIC for something better?
Uhm... it's for beginners. We can't ditch the biginners...

Isn't it about time we ditched SCO Unix for something better?
Linux... we think.

Isn't it about time we ditched DOS for something better?
Windows XP

Isn't it about time we ditched Dubya for something better?
John Kerry

Re:think about that sentence:

Plus, you get mascot that is on par with barny for lamenes!

I'll take an overweight penguin instead of an idiotic man in a butterfly suit any day...

Re:think about that sentence:

Problems solved:

Isn't it about time we ditched FTP for something better? HTTP
Isn't it about time we ditched floppy disks for something better? Tape drives
Isn't it about time we ditched IDE drives for something better? Cool, thin IDE cables
Isn't it about time we ditched x86 for something better? x86-64
Isn't it about time we ditched Microsoft Windows for something better? Windows XP
Isn't it about time we ditched CDs for something better? Coasters
Isn't it about time we ditched telnet for something better? Clear text passwords over HTTP
Isn't it about time we ditched CRTs for something better? Incandescent light arrays
Isn't it about time we ditched 20-year-old TV sets for something better? 19 year-old TV sets
Isn't it about time we ditched COBOL for something better? FORTRAN
Isn't it about time we ditched BASIC for something better? Power BASIC
Isn't it about time we ditched SCO Unix for something better? SCO Linux
Isn't it about time we ditched DOS for something better? Protected mode DOS
Isn't it about time we ditched Dubya for something better? Jon Stewart

Re:think about that sentence:

You made your point, but there's a disconnect between your point and why its relevant to the original statement. If someone said to me "isn't it time we ditched floppy disks for something better?" I'd probably say "yeah", not jump all over their ass because someone somewhere uses floppy disks.

piracy returns to ftp?

Interesting... this could bring piracy back to the ftp world, rather than the emule appz or bittorrent world where it's easier to get caught.

Security?

Will there also be an sPDTP, a cryptographically secure version for those who want to secure the set of trusted peers away from the prying eyes of those outside the clique?

binary

If this ever lives on a Sun box, can the default PLEASE not be ascii??? Sun, please get a clue on this!

FTP is the new floppy disk

Even when there is something much better, it will take decades to get rid of it.

about time

Isn't it about time we ditched FTP for something better?"

We already have. It is called SCP [openssh.org]

Re:about time

actually it is called sftp. scp is a substitute for rcp.

P2P Research

There are several P2P research projects that are looking at building reliabale and scalable P2P systems.

Take a look at Tapestry [berkeley.edu], and Chord [mit.edu] (and read some of the papers) to understand the issues involved in providing scalable and high performance P2P services. Not only is scalable search and overlay graph connectivity an issue, but also node failure and short session times of P2P nodes.

Additionally, when you actually handle the issue of downloading data, building application-lvel multicast trees to distribute the data efficiently on a large scale is not easy. Two papers from SOSP '03 SplitStream [microsoft.com], and Bullet [rochester.edu] address that issue.

Mirror

Just in case... here's a mirror. [rightbrainnetworks.com] Always glad to lend a hand.

p2p OS installls

I'm waiting for boot disks that fire up a peer to peer client for installing your os, and updates. Debian would be a great start, it would hugely reduce the load of the servers. Also Fedora, the BSDs, etc.

Yes, you can already do bit torrent for the ISO, but that is its own kind of wast and hassle.

Some day.

Speaking of torrents...

SuprNova [suprnova.org], the best torrent web site ever, is going Japanese.

I swear, this has nothing to do with today's date. :-P

I, for one,

am sick of trying to determine the april fool day jokes from the real stories.

something better == sftp, or HTTP!

I thought something better was sftp. As for distributions.. why not HTTP? Setup one reflector that dynamically kicks outs redirects as new mirrors come online. This is mutch better as we have a ton of clients already installed (curl,wget,..etc) We also have load balancing, dns round robin, authorzation, security(read: SSL) well defined in the protocol. All we need is a cgi script to kick out the redirects, and another that will make signature files based on the publically available SSL cert. Whamo all the same features.. and we didnt have to reinvent the wheel.

Hm...

That's just great! Now the media will consider FTP a movie-stealing method. Then the MPAA will call a ban to all FTP servers!

PDTP's Sketchy Liscensing

I was looking around on the pdtp website. I was thinking everything was fine and dandy until I saw this in the FAQ.

Question:
"Skyfire is using a derivative of the Apache License. Doesn't that preclude linking with Qt as the Apache License is incompatible with the GPL?"

Answer:
"Qt/X11 is dual licensed under both the GPL and the QPL. The Apache License, while incompatible with the GPL, is not incompatible with the QPL, so when Skyfire is linked with Qt/X11 the terms of the QPL apply. Qt Non-Commercial Edition for Windows has a separate set of license terms which apply to all Windows builds of Skyfire." (emphasis added)

The FAQ page [sourceforge.net]

Isn't this license a poor one? Aren't they breaking sourceforge.net rules by using a OSI unapproved license?

Or maybe I don't know what I am talking about. PLEASE Correct me if I am wrong.

QPL is OSI approved

Hello. I'm the project manager for PDTP, and author of Skyfire [sourceforge.net]. There's nothing wrong with the QPL whatsoever, unless you mind the fact that it's GPL incompatible (but then again, so is the Apache license). The QPL is an OSI Approved [opensource.org] license, so there's nothing to worry about.

You can chuck ftp but..

The operating systems are not going to chuck ftp so soon and nor are they going to include torrent as a default program.
I think theres still a while till we ditch ftp and move onto something else completely.Torrents and other p2p stuff is good but only if you take the effort to get them.What about the masses who want to click and go?It won't happen till they can right click and it says "Save torrent as". :-)

There's already a solution that covers this.

HTTP does all that. There are well-defined
and well-implemented (Squid) cache-tree protocols
for HTTP. This is very old stuff. FTP is just
plain obsolete. It ads *zero* value over HTTP,
and it's harder to use. Trying to bring FTP up
to the standards of HTTP is a futile effort too,
since HTTP is mature on many more dimensions,
and does not suffer from the gross defects of
the more primitive FTP such as transmission of
port numbers as stream data.

Re:There's already a solution that covers this.

Actually, there is one thing that is terrible annoying about HTTP, that I always liked about FTP. You can't ask it to enumerate files. Sure it'll give you a list, but you can't just take all the links. They might have custom headers or footers. So you actually have to parse the stupid thing and extract the pieces and parts you want. Every FTP server and client I have ever seen has a scriptable way to say, grab everything in that directory, put it here. HTTP has no such facility.

It's virtually trivial to mirror subparts of an FTP site, it's much harder to do that on a Website if it has any links to the parent. Especially because websites specifically aren't a filesystem. So you can't make the same heirical assumptions that you would about an FTP site. It's why I always use rsync mirrors to grab files instead of FTP or HTTP. I hate FTP, it's a stupid protocol. HTTP is nice, but there is always extra crapola that I don't want that is a part of the system (index files, icon images, other gunk). HTTP isn't a filesystem. Now, WebDAV from what I have seen, looks like it could be a real filesystem. HTTP straight up isn't.

Kirby

Re:There's already a solution that covers this.

It ads *zero* value over HTTP,

HTTP does not do the same things that FTP does.

HTTP sucks for file transfers, frankly. You need a full-fledged web-browser just to view the index of files on an HTTP server. Not to mention that automatically downloading subdirectories requires serious processing of numerous HTML sub-documents.

HTTP does not do a good job of:

handling authentication.

handling sessions.

keeping statistics

limiting connections

communicating error messages
Etc, etc, etc.

does not suffer from the gross defects of the more primitive FTP such as transmission of

port numbers as stream data.

Yes, I think everyone will agree that FTP sucks in that regard, but HTTP has it's own drawbacks.

FTP would disapear quickly if something came along that had all the features of FTP, without the baggage. However, until that something comes along, we are all stuck with FTP.

Terms Of Service

Adelphia.net:

NO SERVERS OF ANY KIND

Sorry, I'd like to participate, but I can't. I know it won't die without me, but I fear more broadband ISPs taking on equally moronic TOS. The stance isn't entirely without merit, since Joe 6pak has no business running a server on today's Internet. But there's also no way to prove competence, and even if there were, I'm sure ISPs would be eager to charge me double.

cool...

the guy at autopackage.org was attempting something simmilar to this but for package distrobution...it looks like with this protocol, youjust need to set up all the OSS servers with packages on them and boom...you have one huge honkin FTP site with all packages nessisary for all things...then you just ned to download a discription file and then the package manager can grab all the packages from a few PDPT gets and your done...good bye RPM hell.

BitTorrent resource-hungry?

From the description:
BitTorrent suffers another problem in that the only usable implementations are currently only available in Python. The primary problem with Python is its excessive resource usage
Really? I'm currently running four throttled BT downloads on a PII-350 w/64MB. Max CPU usage is 8%, load average 0.25. If you're really that bothered see here [sourceforge.net] for an alternative.
but other problems arise such as integration of the Python implementation into a native GUI frontend for a given platform
Ever heard of WxGtk [linux.de]? RPMs for most distros, if it wasn't part of your default install.
as well as the need to bundle the Python runtime with the BitTorrent client on most platforms as few deployed systems have a Python runtime available
Now this is just silly. I dont think there is a linux distro which doesn't include Python libraries and even for Windows it's a single small executable. Besides (correct me if I'm wrong) but isn't one of the reasons for using Python that it has bounds-checking on arrays and is therefore proof against the cause of most exploits - the buffer overrun?

WE already have replaced FTP

There's ssh and WebDAV. Both work great, and have significant advantages...

Upgrade yes replace no

The main problem with the "BitTorrent" idea is that it gets associated with "illegal" actions.
I was on the "Desert Combat" Testers team and we had to download 600-700mb patches once a week... off one ftp server.
When i mentioned the idea of using a modified BitTorrent client/server to ease the strain on the server i was told we could not use "illegal" tools.

First educate the public and then start to think about upgrading things to help the internet not crash and burn.

Phil

BT Bandwidth-saved?

Hmm, I was thinking about this earlier.. does anyone actually have any statistics for how much server transfer badwidth was saved by distributing a popular file (latest anime release or something) over BitTorrent? How much does it actually help?

Forward error correction and bulk data transfers

What I'd like to see incorporated in their protocols is the proper use of forward error correction techniques to speed up file tranfers. Most Internet file transfer protocols these days make use of automatic rerequest to correct mistransmitted packets, which can be costly when large amounts of data are to be transferred over unreliable links. I've heard that it is possible to make use of erasure codes such as Reed-Solomon or Tornado codes [berkeley.edu] to speed up multicast bulk data transfers. Michael Luby [berkeley.edu] the inventor of the Tornado codes, has actually considered this application for his error correcting codes in the paper "Accessing Multiple Mirror Sites in Parallel: Using Tornado Codes to Speed Up Downloads [berkeley.edu]". I wonder if this approach would help improve this application's performance?

i used to use ftp all the time..until...

i got samba correctly configured, a while ago.
recently i've reformatted my box and freshly installed debian, and for some reason, when i attempt to transfer files via ftp, or samba, the system locks up instantly. no indication of why in /var/log/ either. what a bummer.
kernel 2.6.4. ipv6 is not enabled.

FTP vs TELNET/SSH

"Isn't it about time we ditched FTP for something better?"

Yes, it is. However, SSH has been around for a significant time and still hasn't replaced telnet, even given the horrific security holes in telnet.

Re:Sigh....

Heh...a few years ago, /. made an April fools joke about Python and Perl merging into a new language called "Parrot" Apparently, some people liked the idea, and started the project. I have no idea of its status, though :-(

Re:Sigh....

Currently at v0.1.0 [parrotcode.org], awaiting Something Big in Perl 6, it would seem.

Re:joke?

Read the "from the"... department

Re:It still doesn't answer a very important questi

Answer to this is the same argument that I've heard sometimes applied to open source:

If we all contribute a little, then the cost to all of us is that much less.

Re:It still doesn't answer a very important questi

There's no problem with that, don't share your bandwith with anyone, noone shares bandwith with you. Then you can only download from one source with limited resources. Other people that share can download from many sources (eventually each one with much less resources) that provide a total bandwith much greater, and more, when there is more people downloading they also download faster, instead of you that don't wanna share and have to slower the download when more people that don't share start to download... And besides, most people's connection limits are dowload limits not uploads :-P

Re:It still doesn't answer a very important questi

I, for one, will not be sharing my bandwidth for something that should be provided by the file's source

So, if for example, I write this need little GPL'd app that everyone loves, and release it as opensource, I should be responsible for hosting the file server for everyone? What if hundreds of thousands of people use it everyday, and a new patch comes out. Should I have to buy a T-1 (or something bigger) that costs an arm and a leg, to provide the file patch for a free program to others with no income for me? Or should I ask others to help out with their extra bandwith, and get a few seeders out there with bittorrent and run the tracker with the DSL line i have. I could pay $20 a month for a metered tiered connection in my town, but I pay $50 for an "unlimited" (notice the quotes). I know that not everywhere has these kinds of services, but you don't have to leave the torrent open forever either, or just leave the upload at 1k/s or something. It might slow down your download, but your still going to get access to the file..

Re:It still doesn't answer a very important questi

and you, for one, maybe don't deserve the bandwidth you get for _free_ downloads. your favorite linux distro has to pay the bills sometime too, and i don't imagine that you've actually paypal'd anyone to host their isos. so contribute (either by cash or bandwidth) or shut up.