Difference between revisions of "Metalink"

Revision as of 01:06, 1 December 2006

Metalink is an open standard/framework for programs that download (download managers, BitTorrent clients, Web browsers, FTP clients, & P2P programs). For increased convenience, it stores the many locations of files (FTP/HTTP/P2P) in a single XML file (a .metalink) so chunks/segments of each file can be downloaded from multiple resources at the same time (known as accelerated/multi-threaded/segmented downloading). If you have a fast Internet connection, this usually results in a more reliable and much faster download than a typical single segment download from one server. Other features include automatic checksum comparison of finished downloads, and automatic OS/language selection. Metalinks are also backwards compatible with regular hyperlinks by using the format http://URL#!metalink3!http://URLtoMetalink.

Metalink is extensible, and supports listing checksums (frequently MD5 & SHA-1) along with PGP signatures. Besides FTP and HTTP mirror locations and rsync, it also supports listing these P2P methods: BitTorrent, ed2k, and magnet link.

Here's how it works: One link, one click downloads for all people. For instance, a Metalink for the OpenOffice.org installation files can list every OS and language combination and their mirror locations and checksums. Someone in Brazil that speaks French and uses Linux can click on the link & get the French Linux version from the local Brazilian mirrors. Someone in Japan that speaks English on Mac OS X can click that same link & get the English OS X version from the local Japanese mirrors. Once the download finishes, the checksum is automatically verified, without user interaction. (This is available now, but not all clients function this way by default).

Client programs

• aria2 (GPL) is a Unix command line "high speed download utility" that supports BitTorrent and Metalink (HTTP/FTP/BitTorrent).
• FlashGot (GPL), a Firefox extension for integrating with many Download Managers, supports Metalink.
• GetRight 6, a Windows Download Manager and BitTorrent client, supports Metalink (HTTP/FTP/BitTorrent).
• Speed Download, a Mac Download Manager, supports Metalink (HTTP/FTP)as of version 4.1.0.
• wxDownload Fast (GPL) is a Mac/Unix/Windows download manager that supports Metalink (HTTP/FTP).

See a Comparison of download managers for details on which of them support Metalink and have other features.

In use

Currently, OpenOffice.org uses Metalinks on an alternate P2P download page with Magnet links.

The following distributions use Metalink for ISO distribution: Arch Linux, DesktopBSD, PC-BSD, BLAG Linux and GNU, StartCom Enterprise Linux, Berry Linux, Ubuntu Christian Edition, PuppyLinux, redWall Firewall.

It is also being tested with Arch Linux's Pacman which is used for package management.

Metalink generation

Metalink is XML so it can be hand crafted or XML tools can be used for automation. These tools are specifically for Metalink.

• RoPkg::Metalink / Metalink @ Packages Resources (GPL) for automated Metalink generation.

• Metalink Generator can also be used to create Metalinks by filling out a form.

• Metalink tools (GPL) Unix/Windows command line tools for generating Metalinks with checksums and mirror/p2p lists.

• Simba (GPL), the mirroring tool, includes automatic Metalink generation as one of its features.

Example .metalink file

.metalinks are XML text files.

<metalink version="3.0" xmlns="http://www.metalinker.org/">
  <files>
    <file name="example.ext">
    <verification>
      <hash type="md5">example-md5-hash</hash>
      <hash type="sha1">example-sha1-hash</hash>
      <signature type="pgp"/>
    </verification>
    <resources>
      <url type="ftp" location="us" preference="90">ftp://ftp.example1.com/example.ext</url>
      <url type="ftp" location="uk" preference="90">ftp://ftp.example2.com/example.ext</url>
      <url type="http" location="us" preference="90">http://www.example1.com/example.ext</url> 
      <url type="http" location="uk" preference="90">http://www.example2.com/example.ext</url>
      <url type="http" location="de" preference="90">http://www.example3.com/example.ext</url> 
      <url type="bittorrent" preference="100">http://www.ex.com/example.ext.torrent</url>
      <url type="magnet"/>
      <url type="ed2k"/>
    </resources>
    </file>
  </files>
</metalink>

Future Use

Possible areas of future investigation for OpenOffice.org:

• More testing on download pages.
• "Check for Updates"/Autoupdate feature in OpenOffice.org 2.0.4+.
• Integration with Bouncer and a Firefox extension like DownThemAll.

Press descriptions

"Downloading bliss with Metalink" by Mayank Sharma:

"Getting popular software off the Internet can sometimes be a struggle, even with all the mirrors and BitTorrent Samaritans out there. When the Fedora project released Fedora Core 6 last month, for instance, even several dozen mirrors weren't enough to serve everyone, and torrent speeds weren't good enough because of a scarcity of seeders. But thanks to Metalink I was able to sleep while my FC6 ISOs were downloading.

Metalink is an open standard that claims to make downloading easier, faster, and more reliable by helping users extract the last drop of juice out of their connection. But Metalink isn't your run-of-the-mill download accelerator. It is in fact a framework for use by other download clients, and bundles traditional HTTP and FTP methods of downloading files along with BitTorrent.

The Metalink standard replaces static URLs with a .metalink file, which is a simple XML file. The file in turn contains locations of all the mirrors of the application you want to download. In addition to HTTP and FTP mirrors and rsync, Metalink supports several P2P methods as well, including BitTorrent, ed2k, and magnet links. For example, the OpenOffice.org metalink contains links to more than 50 HTTP and FTP servers and a torrent.

If I were using a regular hyperlink to download OpenOffice.org and the server went down midway through the transfer, I would be left with an incomplete transfer that could either be resumed or not depending on the download client. But with Metalink, if one server goes down, the client software simply jumps to another mirror and resumes downloading from that point. In effect, this increases reliability, as all listed servers would need to be down for the file to be unavailable.

Files downloaded with metalinks are automatically verified. Metalink supports both MD5SUM and SHA1SUM checksums as well as PGP signatures, which are embedded in the .metalink file itself. Checksums are unique for every file, like a fingerprint. If there's an error in transfer, or if someone has maliciously replaced a good file with a bad one, the checksums won't match. If that happened with another file transfer method, you'd have to download the file again from another mirror. With Metalink, if the application you are downloading has a torrent, Metalink can use a torrent's partial file or chunk checksums to verify mirror downloads as well. If only a small chunk of a download has errors, Metalink just re-downloads that part instead of the whole file."

OSresources article:

"Metalink is a system which improves the download process by increasing availability and guaranteeing integrity. It can give your users a more reliable download by providing multiple links to the same file, which can be switched to if one server is down or fails during transmission. It can also make downloads faster by using multiple resources at once. Organizations use mirror servers in multiple locations to provide redundancy and place content nearby users in the hopes that it will be faster for local users. Examples would be mirrors on each continent or near a concentration of users. Downloaders in Europe would manually choose a mirror located in a European country, preferably their own, and people in Japan or other countries would choose mirrors close to them. If mirrors are down they can be used by downloaders in other locations as a backup or second option. Metalink lists mirrors with machine readable information on priority and location so their efficient use can be automated by download programs. It can list mirrors around the world, but will automatically default to mirrors closer to you and by priority.

Download programs (also known as download managers) that are Metalink-aware read the information from a Metalink file (one with a .metalink extension), which is just plain XML. The download programs split the file tranfer into independently verifiable segments which are re-assembled. By downloading multiple segments from multiple servers, download speeds are usually three to four times faster on an average broadband connection and much faster on business grade connections. The checksum verification process, usually manual and arcane to most people, is automated with Metalink, so files are guaranteed to be an exact copy of the file you downloaded, free of errors. Metalinks can also contain publisher information, Operating System and architecture, language, file descriptions, mutliple files (to be added to a download queue) and so on. All this extra information allows download programs to do interesting things. It lets you have one download link for multiple Operating Systems and languages, and downloaders will get the correct version for their system."

TorrentFreak explains Metalink like this:

"We’ve all seen those download pages where a couple of http, ftp and bittorrent mirrors are listed. We often just pick one of these, but wouldn’t it be great if you could use them all at the same time? Metalink has the solution. It works like this, the .metalink file tells your download manager where it can get the file, and connects to these different sources at the same time."

downloadsquad describes it this way:

"Metalinks make complex download pages obsolete by replacing long lists of download mirrors and BitTorrent trackers with a single .metalink file. As you might have already guessed, a .metalink file is a file that tells a download manager all the different ways it can download a file. The file itself takes the form of an open XML standard that can list an unlimited number of HTTP and FTP sources as well as BitTorrent trackers and ed2k and magnet links."

External links

• Metalink homepage
• Metalink @ Packages Resources : Automatically generated Metalinks for Open Source software along with source code for the Metalink generator.
• Simba (GPL), the mirroring tool.
• OpenOffice.org P2P page