Metalinks Tries to Simplify Downloads 62
ant_tmwx writes "Metalinks collect information about files in an XML format used by programs that download. The information includes mirror lists, ways to retrieve the file on P2P networks, checksums for verifying and correcting downloads, operating system, language, and other details. Using Metalinks details the Free Software programs you can use to download them with. There are also clients on Mac and Windows. With a list of multiple ways to download a file, programs can switch to another method if one goes down. Or a file can be downloaded from multiple mirrors at once, usually making the download go much faster. Downloads can be repaired during transfer to guarantee no errors. All this makes things automatic which are usually not possible or at least difficult, and increases efficiency, availability, and reliability over regular download links. OpenOffice.org, openSUSE, and other Linux/BSD distributions use them for large downloads."
Mirror? (Score:2)
Or, depending on your location, use one of these mirrors:
http://uk.exampleURL.com/file.metalink
http://nl.exampleURL.com/file.metalink
http://de.exampleURL.com/file.metalink
Seriously though, I like the basic idea, but the system does add an extra point of failure.
Re: (Score:1)
> [...] I like the basic idea, but the system does add an extra point of failure.
>
Avoiding problems with dead servers is not the only use of mirrors. Metalink is more used to group all informations to get a file, reliably (hash checks), using your preferred protocol (and, possibly, multiple protocols at once), and as fast as possible (by downloading little, from a lot of places, rather than a lot, from a single place).
Re: (Score:2)
Does it? Its not totally fool proof since the original server hosting the link file could still go down, but thats not an extra point of failure, since its no different then the server going down when trying to download the real file from it in the first place. Once you have the Metalink file everything should be fine.
Browser clients available? (Score:5, Insightful)
It's bad enough when I tell my dad to download a torrent and he complains that a torrent manager client pops up; especially when he doesn't realize that closing the window may not stop the torrent.
Re:Browser clients available? (Score:4, Informative)
GetRight (Windows) is a Download Manager that supports Metalink.
FlashGot (Cross platform, Open Source) is a Firefox extension that integrates with around 30 Download Managers and supports Metalink with wxDownload Fast, Speed Download, and GetRight 5.2d and later.
aria2 (Unix/Windows, Open Source) is a high speed download utility that supports segmented downloads, BitTorrent, and Metalink (HTTP/FTP/BitTorrent integrated) from the command line.
Speed Download (Mac) is a slick Download Manager with fast downloads & P2P filesharing. It integrates with popular Mac browsers like Safari, Camino, & Firefox (along with 5 others).
wxDownload Fast (Mac/Unix/Windows, Open Source) supports Metalink.
Free Download Manager (Windows) unreleased BETA supports Metalink.
Orbit Downloader (Windows) is a new download manager with interesting P2P features.
SmartFTP (Windows), an excellent FTP client on Windows, supports Metalink for adding files to a transfer queue and checksum verification (no acceleration).
Phex (Mac/Unix/Windows) is a Gnutella P2P client that can export Metalinks.
If you like Metalink, request support in these clients on their forums or bug trackers:
CuteFTP is a Windows GUI FTP client.
Bouncer (Open Source) powers downloads for various projects like Mozilla. A patch for Metalink generation has been submitted by Bram Neijt but not yet integrated.
cURL (Cross platform, Open Source) an interface for libcurl would be cool. Anyone up for writing one?
Shareaza (Open Source) would be great as well.
Opera is a great standards compliant browser that's always adding new features. Current threads: 1, 2, 3
Firefox (Open Source) is pretty neat too.
Re: (Score:2)
# Free Download Manager (Windows) unreleased BETA supports Metalink.
# Orbit Downloader (Windows) is a new download manager with interesting P2P features.
Jonah HEX
Re: (Score:1)
KGet for KDE4 (their download manager) also has just added initial support.
Both are free open source projects, so help out if you're able.
Re: (Score:1)
Gee willakers! (Score:2)
I don't think it's a bad idea, I just don't get why it's on
Or in the case of us gentoo folk, just build the damn program and eliminate any doubt
Tom
Re: (Score:2)
Needless extra step? (Score:3, Insightful)
Why should you use it?
Users
Your downloads will be simpler, faster, and more reliable...without you doing anything differently.
Bittorrent already does this just about as effectively as this idea will.
Developers
It's a neutral framework that doesn't favor any one program, Operating system, or group, and is easy to implement.
Once again, bittorrent is just as easy. And its OS agnostic.
Site owners
Resume and recover from single servers going down.
Sorta an issue with bittorrent, but not really. House the seed in multiple locations. Or better yet, have your clients take a copy of the seed and share that with their peers in the case of a downed server.
Downloads can automatically be split between sources (mirrors/P2P) and all downloads will be verified.
More people can get access to your files easier, more reliably, even at the most heavily accessed times.
This means less retries and cheaper bandwidth and support bills. Saving money = good.
Once again, this is where bittorrent shines. A lot of people going after your files? Great, that means it's got a better availability on your torrent, more bandwidth for everyone.
To me, this looks like a solution in search of a problem.
Re: (Score:3, Insightful)
Bittorrent is neat and handy if you don't have the capacity or balls to exclusivel
Re:Needless extra step? (Score:5, Interesting)
This system, however, would allow me to easily download automatically from the mirror closest to me. It would also be pretty easy using a little ECMAScript to grab the correct mirror using the browser's locale and present a direct download link.
[1] Actually, it's a pretty horrible protocol. It doesn't do anything with location information, so it adds a lot of extra load to the backbones. It uses TCP, so it will never support multicast without a fundamental redesign, and the algorithms for rate distribution have some fundamental flaws.
Your comment doesn't sound right to me... (Score:4, Interesting)
And I'm really asking here. Not just disguising my attacks as questions.
Re: (Score:2)
Really, I've always prefered downloads split between multiple fast mirrors(split http, FTP as done by the actual piracy scene) rather than multiple slow client peers (utorrent, kazaa, any other common 'end user' piracy). I hope this metalin
Re: (Score:2)
Re: (Score:1)
Re: (Score:1)
Re: (Score:2)
It also depends on your definition of "fast" really. If the files ETA is more than 15 minutes I'm usually going to spend at least the first 5 looking for a better way to download it.
Re:Who says it can't use BitTorrent also? (Score:5, Insightful)
Example - MetaLink XML contains the following formats:
5 different HTTP sites
2 FTP sites
3 BitTorrent Trackers
eMule/Edonkey Hash
Example - Client One has implemented:
HTTP, FTP and BitTorrent
Example - Client Two has implemented:
BitTorrent and eMule
Example - Client Three has implemented:
HTTP, FTP, BitTorrent and eMule
I'm surprised it's taken this long to come up with this sort of client independant format.
Jonah HEX
Re:Needless extra step? (Score:5, Insightful)
If the tracker goes down or there happens to be a lack of seeds the link is dead. With Metalink on the other side a client could automatically use *all* ways to get a file, not just a single tracker or server, but multiple http servers, P2P networks and torrent all at the same time, if one goes down there might still be plenty of others left.
### Once again, bittorrent is just as easy. And its OS agnostic.
But not protocol or server agnostic.
### To me, this looks like a solution in search of a problem.
Ever tried to download a file from Sourceforge or any other server with a dozens of mirrors which you have to manually select? That is exactly the problem that Metalink solves, its a standard way to show where the mirrors are, completly independent of the protocol in use. Thus it allows the client to automatically select them or use them all at the same time for faster download, no more stupid manual mirror selection just to find out that the host is down or slow as hell.
Metalink doesn't try to replace bittorrent, quite the opposite, it tries to provide a way to simply bundle all links that lead to the same file, torrent included.
Re: (Score:2)
With DHT and multiple trackers a torrent is pretty much unkillable. Even if you have a lot of peers and no seeds, you usually have the entire file distributed throughout the peers and more seeds will be created.
For example on isohunt.com each torrent has maybe 5 (plus or minus) trackers associated with it. And DHT ensures that even if ALL of the trackers go down, peers will continue to find and download from each other.
It is a fairly robust download met
Re: (Score:2)
Power users w
It exists ! (Score:2)
And it exists. Either...
Fuck BitTorrent. (Score:2)
Re: (Score:2)
Not news (Score:2)
Re: (Score:2)
I can't see how this is a bad idea, it seems like great to me. Sure bittorrent has its place, but so does this.
And what is wrong with you? Why can't we talk about this idea? Maybe you are on the wrong site, fox news is maybe more what you want.
YUM, Sourceforge, kernel.org, ... (Score:2)
Re: (Score:2)
Mac client confusion and Open Standard (Score:2)
Unrelated, I saw numerous attempts of such integrated p2p downloads. The part that got me from the Metalink ma
Mac client confusion: ok, I get it now... (Score:2)
Location embedding? (Score:2)
Re: (Score:3, Interesting)
What did I got wrong?
You are not thinking in terms of cost. When you buy capacity in bulk, you might get a flat rate, but you are more likely to get one rate for national and one rate for international traffic. In the UK, I believe it's fairly common to get a more expensive rate for transatlantic traffic. If I'm hosting a mirror in the UK, then it will cost me more if Americans use it than people in France. Any finer-grained information doesn't really matter.
Of course, with IPv6 we ought to be able to get rid of this kind
Re: (Score:1)
Specifications? (Score:2, Insightful)
Re: (Score:1, Informative)
http://www.metalinker.org/Metalink_3.0_Spec.pdf [metalinker.org]
Re: (Score:1)
Re: (Score:1)
good idea - can be improved (Score:4, Informative)
metalink://host.com/file.ml
Then inside file.ml simply a list of URLs and weights...
ftp://host1.com/file.rpm [host1.com] 10
http://host2.com/file.rpm [host2.com] 10
torrent://host3.com/file.rpm 20
etc
XML doesn't help.
Re: (Score:2)
Call me sceptical (Score:1)
Re: (Score:2)
Metalinks Tries to Simplify Downloads? (Score:2)
That seems like a logical growth of Bittorrent.
Trying to figure out exactly what is needed though was another matter. After a half hour and three or four web sites I wound up with the wxDownload Fast [sourceforge.net] Windows download manager and a Metamirrors Firefox plugin [infosnel.nl].
Is it all working as advertised? Well, stuff is downloading (OpenSUSE 10.2) but I have no idea of it's faster, o
Re: (Score:1)
Not sure I understand what kind of checklist you want. That a client is needed ought to be fairly obvious. Your "then do THIS" depends entirely on the client.
You can 'apt-get install aria2' for a cmdline utility in the style of wget, supporting metalinks.
Re: (Score:2, Informative)
Re: (Score:1)
Faster. Really? (Score:2)
M users simultaneously download from N servers individually. Or M users simultaneously download from N servers in parallel. Aside from some load balancing, can you really gain dramatically? The same number of bytes must be transfered from the same number of servers.
Re: (Score:2)
Yes, in most cases. The bottleneck of a connection is usually near the downloader's end. If such a bottleneck segment is shared by several people, then downloading in parallel (from several servers or even from the same server) helps: the downloader will receive a bigger share of the bottleneck segment.
See here [mail-archive.com] for a little experiment that illustrates the point and that you can reproduce easily.
Re: (Score:2)
It works as long as you are the only one doing it, but when everyone starts using segmented downloads, everyone loses. And you not only
go back to as it was before, you also manage to waste more than the same number of poeple would have before, if they would stick to using
one connection per download.
There has been discussions on the openoffice distribution lists about this to
Metalinks are indeed awesome (Score:1)