A couple of years ago i sketched out something like this for a project at work.
It was with a known cloud of clients though, so security could easily be beefed out with no concerns to "compatiblity" on the client side.
Basically, a big ol tracker running in position X.
A number of headless clients connected to storage systems, spread across the world, potentially divided into a hierarchy based on connectivity.
Each client uses a unique key-pair to communicate with the tracker (phase 1 of security, for a 3rd party it'd be a bitch to sniff, not even another client could sniff and make sense of any of the data related to the first client).
In client communications, a random (weaker) cipher is used, to obscure the data transfer slightly more.
So basically, heavily encrypted tracker communication, weaker encrypted client communication.
All client->client communication apart from the actual data channel also went via the tracker in question, so the only client->client happenings would be a port opening on client A and a connection request from client B, handshake and then the data transfer.
At the time of new files, a "Hello. Wake up now omgplz"-kind of request would be made available to the clients, starting with the innermost ones first (to facilitate a relatively speedy distribution of the files across the entire network, while attempting to not completely clog the intarweb tube of the "master client"), and when deemed appropriate (x% completion among y% of tier-z clients), move on to tier-(z+1).
This, of course, won't scale to something the size of pirate bay without some serious hardware from the future, but i find it an interesting sidetracking :p