Metalinks Tries to Simplify Downloads

ant_tmwx writes "Metalinks collect information about files in an XML format used by programs that download. The information includes mirror lists, ways to retrieve the file on P2P networks, checksums for verifying and correcting downloads, operating system, language, and other details. Using Metalinks details the Free Software programs you can use to download them with. There are also clients on Mac and Windows. With a list of multiple ways to download a file, programs can switch to another method if one goes down. Or a file can be downloaded from multiple mirrors at once, usually making the download go much faster. Downloads can be repaired during transfer to guarantee no errors. All this makes things automatic which are usually not possible or at least difficult, and increases efficiency, availability, and reliability over regular download links. OpenOffice.org, openSUSE, and other Linux/BSD distributions use them for large downloads."

Mirror?

http://www.exampleURL.com/file.metalink

Or, depending on your location, use one of these mirrors:
http://uk.exampleURL.com/file.metalink
http://nl.exampleURL.com/file.metalink
http://de.exampleURL.com/file.metalink

Seriously though, I like the basic idea, but the system does add an extra point of failure.

Browser clients available?

Are there clients that integrate (ie: extensions) for Firefox, IE, Safari, and Opera? If there is proper integration with these clients (meaning seamless downloading without opening third party download managers), this might actually go well.

It's bad enough when I tell my dad to download a torrent and he complains that a torrent manager client pops up; especially when he doesn't realize that closing the window may not stop the torrent.

Re:Browser clients available?

From http://www.metalinker.org/implementation.html [metalinker.org]:

Download Metalinks with these programs...
GetRight (Windows) is a Download Manager that supports Metalink.
FlashGot (Cross platform, Open Source) is a Firefox extension that integrates with around 30 Download Managers and supports Metalink with wxDownload Fast, Speed Download, and GetRight 5.2d and later.
aria2 (Unix/Windows, Open Source) is a high speed download utility that supports segmented downloads, BitTorrent, and Metalink (HTTP/FTP/BitTorrent integrated) from the command line.
Speed Download (Mac) is a slick Download Manager with fast downloads & P2P filesharing. It integrates with popular Mac browsers like Safari, Camino, & Firefox (along with 5 others).
wxDownload Fast (Mac/Unix/Windows, Open Source) supports Metalink.
Free Download Manager (Windows) unreleased BETA supports Metalink.
Orbit Downloader (Windows) is a new download manager with interesting P2P features.
SmartFTP (Windows), an excellent FTP client on Windows, supports Metalink for adding files to a transfer queue and checksum verification (no acceleration).
Phex (Mac/Unix/Windows) is a Gnutella P2P client that can export Metalinks.

...

If you like Metalink, request support in these clients on their forums or bug trackers:

CuteFTP is a Windows GUI FTP client.
Bouncer (Open Source) powers downloads for various projects like Mozilla. A patch for Metalink generation has been submitted by Bram Neijt but not yet integrated.
cURL (Cross platform, Open Source) an interface for libcurl would be cool. Anyone up for writing one?
Shareaza (Open Source) would be great as well.
Opera is a great standards compliant browser that's always adding new features. Current threads: 1, 2, 3
Firefox (Open Source) is pretty neat too.

So looks like we need to wait alittle.

Gee willakers!

You can do that with computers? Honestly, how hard is it to pick your OS out of a list of download links?

I don't think it's a bad idea, I just don't get why it's on /.

Or in the case of us gentoo folk, just build the damn program and eliminate any doubt :-)

Tom

Needless extra step?

From their page:

Why should you use it?

Users
Your downloads will be simpler, faster, and more reliable...without you doing anything differently.

Bittorrent already does this just about as effectively as this idea will.

Developers
It's a neutral framework that doesn't favor any one program, Operating system, or group, and is easy to implement.

Once again, bittorrent is just as easy. And its OS agnostic.

Site owners
Resume and recover from single servers going down.

Sorta an issue with bittorrent, but not really. House the seed in multiple locations. Or better yet, have your clients take a copy of the seed and share that with their peers in the case of a downed server.

Downloads can automatically be split between sources (mirrors/P2P) and all downloads will be verified.
More people can get access to your files easier, more reliably, even at the most heavily accessed times.
This means less retries and cheaper bandwidth and support bills. Saving money = good.

Once again, this is where bittorrent shines. A lot of people going after your files? Great, that means it's got a better availability on your torrent, more bandwidth for everyone.

To me, this looks like a solution in search of a problem.

Re:Needless extra step?

BitTorrent has two major disadvantages:

1. It is fundamentally inefficient. The overall overhead of using BitTorrent on the Internet is greater than direct downloads.[1]
2. It uses the most expensive bandwidth for distribution. Consumer upload bandwidth is the most scarce and about the most expensive bandwidth you can buy

The only advantage of BitTorrent is that you (the content provider) aren't paying for as much of the bandwidth. On the other hand, bandwidth bought in bulk is so cheap that it's really not worth if for most people.

This system, however, would allow me to easily download automatically from the mirror closest to me. It would also be pretty easy using a little ECMAScript to grab the correct mirror using the browser's locale and present a direct download link.

[1] Actually, it's a pretty horrible protocol. It doesn't do anything with location information, so it adds a lot of extra load to the backbones. It uses TCP, so it will never support multicast without a fundamental redesign, and the algorithms for rate distribution have some fundamental flaws.

Your comment doesn't sound right to me...

If it's a pretty horrible protocol, why has it become so popular? Is it possible that all or some of those design "flaws" you mentionned are necessary for it to do what it does? And if not, why have you not created your own, awesomer fileswarming protocol? In fact, why has nobody else?

And I'm really asking here. Not just disguising my attacks as questions.

Re:Who says it can't use BitTorrent also?

It does not look like it excludes ANY type of file transfer, if your client supports it you can do it is how it looks to me.

Example - MetaLink XML contains the following formats:
5 different HTTP sites
2 FTP sites
3 BitTorrent Trackers
eMule/Edonkey Hash

Example - Client One has implemented:
HTTP, FTP and BitTorrent

Example - Client Two has implemented:
BitTorrent and eMule

Example - Client Three has implemented:
HTTP, FTP, BitTorrent and eMule

I'm surprised it's taken this long to come up with this sort of client independant format.

Jonah HEX

Re:Needless extra step?

### Bittorrent already does this just about as effectively as this idea will.

If the tracker goes down or there happens to be a lack of seeds the link is dead. With Metalink on the other side a client could automatically use *all* ways to get a file, not just a single tracker or server, but multiple http servers, P2P networks and torrent all at the same time, if one goes down there might still be plenty of others left.

### Once again, bittorrent is just as easy. And its OS agnostic.

But not protocol or server agnostic.

### To me, this looks like a solution in search of a problem.

Ever tried to download a file from Sourceforge or any other server with a dozens of mirrors which you have to manually select? That is exactly the problem that Metalink solves, its a standard way to show where the mirrors are, completly independent of the protocol in use. Thus it allows the client to automatically select them or use them all at the same time for faster download, no more stupid manual mirror selection just to find out that the host is down or slow as hell.

Metalink doesn't try to replace bittorrent, quite the opposite, it tries to provide a way to simply bundle all links that lead to the same file, torrent included.

Not news

If this standard is accepted by IE and Firefox, *then* it will be news. Until then it's just buzzword compliant crap.

YUM, Sourceforge, kernel.org, ...

It will be great for sites that have a LOT of mirrors. Particularly sourceforge, of which no single mirror is reliably "fast enough" for me.

Mac client confusion and Open Standard

Strangely enough, the OpenOffice distribution page links to a pay-for Metalink client for the Mac [openoffice.org], but hopefully, just two links above, there's the cross-platform open source client. That's confusing. I clicked on the Mac client, thinking I would end up at a Mac-oriented free (even open source?) client, but no, one must choose the cross-platform. Nothing really wrong there, only that it's confusing.

Unrelated, I saw numerous attempts of such integrated p2p downloads. The part that got me from the Metalink main page: "Metalink is an Open Standard". This makes me believe I will join the bandwagon. And yup, the Wikipedia Metalink page is (surprisingly?) informative [wikipedia.org].

Location embedding?

I tried to find additional info about location embedding but haven't succeeded so far. From the wikipedia page [wikipedia.org], the XML code includes url type="http" location="uk" preference="90">http://www.example2.com/example.ex t /url, the part I don't get, as a geospatial professional (see sig), is why the location is encoded as the country code. What are the reasons? Does it make more sense to encode location with a simple lat-lon values (similar to, say, the georss [georss.org] standard do). Some countries being so large, I fail to see the country as a good indicator of distance between computers. What did I got wrong?

Specifications?

I've looked around the site, and I've found no document for the specification of a metalink file. IMO, this will easily lead to many conflicts with different clients each having their own version of "Metalinks".

good idea - can be improved

This is a good idea. It attempts to formalize something thats been done many times before. We do it manually when download from Sourceforge, Yum has a list of mirrors it does automatically. A standard would be nice. I would like to see a new web protocol for it - ie:

        metalink://host.com/file.ml

Then inside file.ml simply a list of URLs and weights...

        ftp://host1.com/file.rpm [host1.com] 10
        http://host2.com/file.rpm [host2.com] 10
        torrent://host3.com/file.rpm 20
        etc

XML doesn't help.

Call me sceptical

But "simplify" and "XML format" in once sentence does not always "return true;". If the number of information stored in XML will grow how much CPU time and storage it will require? Wouldn't it be better to get in into database and provide XML based API? Querying XML is a bit slower than asking any DB. Even SQLite.

Metalinks Tries to Simplify Downloads?

Well, the summary makes it sound divine - one link, one bit of software that accesses P2P, FTP etc interchangeably to maximize download speed.

That seems like a logical growth of Bittorrent.

Trying to figure out exactly what is needed though was another matter. After a half hour and three or four web sites I wound up with the wxDownload Fast [sourceforge.net] Windows download manager and a Metamirrors Firefox plugin [infosnel.nl].

Is it all working as advertised? Well, stuff is downloading (OpenSUSE 10.2) but I have no idea of it's faster, or even if it's also uploading in P2P fashion.

For God's sake, is it too much to ask that the people behind stuff like this include a simple checklist?

To download using Metalinks you needs
a) A download client (here are links to a few),
b) This Firefox plugin and
c) then do THIS.

Magnet has been pretty much doing this.

It seems like we might as well just use magnet links, since they can include different hash types of the file and different locations to download from. I already have them for certain larger files on my website and work fine if you have a client that supports it.

Faster. Really?

Or a file can be downloaded from multiple mirrors at once, usually making the download go much faster.

M users simultaneously download from N servers individually. Or M users simultaneously download from N servers in parallel. Aside from some load balancing, can you really gain dramatically? The same number of bytes must be transfered from the same number of servers.

Metalinks are indeed awesome

These worked really great for the openSUSE release. Why is it good? It takes out the hassle of having to track down a working/fast mirror. Want to download a large DVD ISO *quickly*? Then this is the way. The small metalink file will have a populated list of around 50 mirrors. The client your using works with making multiple connections to all of these, so you will pretty much ALWAYS MAX OUT YOUR CONNECTION. And still get a really safe download. Why? Because it checks checksums on, IIRC, 4-megabyte chunks.

Reliable and fast, and it spreads out the weight between mirrors, AND it can work with torrents. Metalinks are indeed awesome. Wget and kget support for it is in the works, so stay tuned. :)

Re:Why no segment hashes?

Great question! aria2 [sourceforge.net] is the only client (so far, others are in progress) to support segment hashes (aka chunk checksums/repair information) directly in the metalink. It just came out recently. For an example, check out the opensuse metalinks [metalinker.org]. I think this is one of the most important parts about metalink, files are verified during transfer (and corrupted files are fixed) over plain ftp/http with no server/tracker changes.

Re:Metalink repository?

Metalink @ Package Resources [packages.ro] is the closest thing to a repository. Arch Linux, Centos, Edubuntu, Fedora, Linux Kernel, Kubuntu, OpenOffice.org, opensuse, and Ubuntu metalinks are at http://download.packages.ro/metalink/ [packages.ro]