Difference between revisions of "Metalink"

From Apache OpenOffice Wiki
Jump to: navigation, search
m (Future Use: AU, apt-get for Windows)
(Update)
Line 1: Line 1:
 
:''NOTE: Metalink is unrelated to Oracle Corporation's support website.''
 
:''NOTE: Metalink is unrelated to Oracle Corporation's support website.''
  
'''[http://www.metalinker.org Metalink]''' is an open standard/framework for programs that download (download managers, BitTorrent clients, Web browsers, FTP clients, & P2P programs). For increased convenience, it stores the many locations of files (FTP/HTTP/P2P) in a single XML file (a .metalink) for extra reliability in case one method fails and so chunks/segments of each file can be downloaded from multiple resources at the same time (known as accelerated/multi-threaded/segmented downloading). If you have a fast Internet connection, this usually results in a more reliable and much faster download than a typical single segment download from one server. Other features include automatic checksum comparison of finished downloads, listing multiple files to be added to a download queue, and automatic OS/language selection.
+
'''[http://www.metalinker.org Metalink]''' is a cross-platform and cross-application open standard/framework/file format for programs that download (download managers, BitTorrent clients, Web browsers, FTP clients, & P2P programs). For increased convenience, it stores the multiple download locations for a file (FTP/HTTP/P2P) in a single metafile with the extension <tt>.metalink.</tt> This provides reliability in case one method fails. For fast Internet connections, it also speeds up the process by allowing chunks/segments of each file to be downloaded from multiple resources at the same time (segmented downloading).
  
Metalink is extensible, and supports listing multiple partial and full file hashes/checksums (most clients only support verifying MD5 & SHA-1 though) along with PGP signatures. Besides FTP and HTTP mirror locations and rsync, it also supports listing these P2P methods: BitTorrent, ed2k, and magnet link.
+
Other features include automatic checksum verification of finished downloads, automatic repair of downloads with errors, listing multiple files to be added to a download queue, and automatic OS/language selection.
 +
 
 +
Metalink is extensible, and supports listing multiple partial and full file hashes/checksums along with PGP signatures. Most clients only support verifying MD5 & SHA-1, however. Besides FTP and HTTP mirror locations and rsync, it also supports listing the P2P methods BitTorrent, ed2k, and magnet link.
  
 
Here's how it works: One link, one click downloads for all people. For instance, a Metalink for the OpenOffice.org installation files can list every OS and language combination and their mirror locations and checksums. Someone in Brazil that speaks French and uses Linux can click on the link & get the French Linux version from the local Brazilian mirrors. Someone in Japan that speaks English on Mac OS X can click that same link & get the English OS X version from the local Japanese mirrors. Once the download finishes, the checksum is automatically verified, without user interaction. (This is available now, but not all clients function this way by default).
 
Here's how it works: One link, one click downloads for all people. For instance, a Metalink for the OpenOffice.org installation files can list every OS and language combination and their mirror locations and checksums. Someone in Brazil that speaks French and uses Linux can click on the link & get the French Linux version from the local Brazilian mirrors. Someone in Japan that speaks English on Mac OS X can click that same link & get the English OS X version from the local Japanese mirrors. Once the download finishes, the checksum is automatically verified, without user interaction. (This is available now, but not all clients function this way by default).
Line 10: Line 12:
 
==Client programs==
 
==Client programs==
 
*[http://aria2.sourceforge.net/ aria2] (GPL) is a Unix command line "high speed download utility" that supports BitTorrent and Metalink (HTTP/FTP/BitTorrent).
 
*[http://aria2.sourceforge.net/ aria2] (GPL) is a Unix command line "high speed download utility" that supports BitTorrent and Metalink (HTTP/FTP/BitTorrent).
*[http://www.downthemall.net/ DownThemAll] (GPL) is a Firefox extension download manager that has basic Metalink support in SVN.
+
*[http://www.downthemall.net/ DownThemAll] (GPL) is a Firefox extension download manager that has basic Metalink support.
 
*[http://www.flashgot.net FlashGot] (GPL), a Firefox extension for integrating with many Download Managers, supports Metalink.
 
*[http://www.flashgot.net FlashGot] (GPL), a Firefox extension for integrating with many Download Managers, supports Metalink.
 
*[http://www.freedownloadmanager.org Free Download Manager] (Windows) has support in a BETA version.
 
*[http://www.freedownloadmanager.org Free Download Manager] (Windows) has support in a BETA version.

Revision as of 20:35, 9 July 2007

NOTE: Metalink is unrelated to Oracle Corporation's support website.

Metalink is a cross-platform and cross-application open standard/framework/file format for programs that download (download managers, BitTorrent clients, Web browsers, FTP clients, & P2P programs). For increased convenience, it stores the multiple download locations for a file (FTP/HTTP/P2P) in a single metafile with the extension .metalink. This provides reliability in case one method fails. For fast Internet connections, it also speeds up the process by allowing chunks/segments of each file to be downloaded from multiple resources at the same time (segmented downloading).

Other features include automatic checksum verification of finished downloads, automatic repair of downloads with errors, listing multiple files to be added to a download queue, and automatic OS/language selection.

Metalink is extensible, and supports listing multiple partial and full file hashes/checksums along with PGP signatures. Most clients only support verifying MD5 & SHA-1, however. Besides FTP and HTTP mirror locations and rsync, it also supports listing the P2P methods BitTorrent, ed2k, and magnet link.

Here's how it works: One link, one click downloads for all people. For instance, a Metalink for the OpenOffice.org installation files can list every OS and language combination and their mirror locations and checksums. Someone in Brazil that speaks French and uses Linux can click on the link & get the French Linux version from the local Brazilian mirrors. Someone in Japan that speaks English on Mac OS X can click that same link & get the English OS X version from the local Japanese mirrors. Once the download finishes, the checksum is automatically verified, without user interaction. (This is available now, but not all clients function this way by default).


Client programs

  • aria2 (GPL) is a Unix command line "high speed download utility" that supports BitTorrent and Metalink (HTTP/FTP/BitTorrent).
  • DownThemAll (GPL) is a Firefox extension download manager that has basic Metalink support.
  • FlashGot (GPL), a Firefox extension for integrating with many Download Managers, supports Metalink.
  • Free Download Manager (Windows) has support in a BETA version.
  • GetRight 6, a Windows Download Manager and BitTorrent client, supports Metalink (HTTP/FTP/BitTorrent).
  • KGet (GPL), the native download manager that is part of KDE4, has added initial support to SVN.
  • MetaDL (LGPL) is a NSIS download plugin based on cURL that supports Metalink and downloads from one URL, supports download resume, and verifies checksums. Metadl is used by Wubi, the planned official Ubuntu installer for Windows.
  • Orbit Downloader, a Windows Download Manager, supports Metalink (HTTP/FTP).
  • Phex (GPL), a Mac/Unix/Windows Gnutella client, supports exporting a library to Metalink for sharing with others.
  • SmartFTP, a Windows GUI FTP client, supports Metalink for adding files to an FTP transfer queue and checksum verification (does not supported segmented downloads).
  • Speed Download, a Mac Download Manager, supports Metalink (HTTP/FTP)as of version 4.1.0.
  • wxDownload Fast (GPL) is a Mac/Unix/Windows download manager that supports Metalink (HTTP/FTP).
  • Also, Appupdater (GPL) for Windows "is similar to apt-get or yum on Linux. It automates the process of installing and maintaining up to date versions of programs." It uses Metalinks for a specific set of programs and does not allow downloading with other .metalinks obtained elsewhere.

See a Comparison of download managers for details on which of them support Metalink and have other features.

In use

Currently, OpenOffice.org uses Metalinks on an alternate P2P download page with Magnet links.

cURL, a command line download tool, offers Metalinks of their downloads. UniProt Consortium also provides a Metalink for their large protein database.

The following distributions use Metalink for ISO distribution: openSUSE, Arch Linux, DesktopBSD, PC-BSD, BLAG Linux and GNU, StartCom Enterprise Linux, Berry Linux, Ubuntu Christian Edition, PuppyLinux, redWall Firewall.

Appupdater (GPL) for Windows "is similar to apt-get or yum on Linux. It automates the process of installing and maintaining up to date versions of programs."

It is also being tested with Arch Linux's Pacman which is used for package management.

Metalink generation

Metalink is XML so it can be hand crafted or XML tools can be used for automation. These tools are specifically for Metalink.

  • Bouncer (GPL) is a "database driven mirror management app that has three main components - a bounce script, a sentry script to check mirrors and files for integrity and a set of web-based administration tools." Bouncer supports Metalink via that has not yet been integrated upstream.
  • cURL uses Perl scripts to offer Metalinks on their download pages.
  • Metalink Editor (GPL) is a Cross Platform GUI Metalink editor written in Python which allows you to create and edit Metalinks.
  • Metalink tools (GPL) Unix/Windows command line tools for generating Metalinks with checksums and mirror/p2p lists.
  • MirrorManager (MIT X11 License) is used by the Fedora Project for dynamically listing mirrors. This patch has not yet been integrated upstream.
  • Simba (GPL), the mirroring tool, includes automatic Metalink generation as one of its features.

Example .metalink file

.metalinks are XML text files.

<metalink version="3.0" xmlns="http://www.metalinker.org/">
  <files>
    <file name="example.ext">
    <verification>
      <hash type="md5">example-md5-hash</hash>
      <hash type="sha1">example-sha1-hash</hash>
      <signature type="pgp"/>
    </verification>
    <resources>
      <url type="ftp" location="us" preference="90">ftp://ftp.example1.com/example.ext</url>
      <url type="ftp" location="uk" preference="90">ftp://ftp.example2.com/example.ext</url>
      <url type="http" location="us" preference="90">http://www.example1.com/example.ext</url> 
      <url type="http" location="uk" preference="90">http://www.example2.com/example.ext</url>
      <url type="http" location="de" preference="90">http://www.example3.com/example.ext</url> 
      <url type="bittorrent" preference="100">http://www.ex.com/example.ext.torrent</url>
      <url type="magnet"/>
      <url type="ed2k"/>
    </resources>
    </file>
  </files>
</metalink>

Future Use

Possible areas of future investigation for OpenOffice.org:

  • More testing on download pages.
  • "Check for Updates"/Autoupdate feature in OpenOffice.org 2.0.4+. Appupdater checks for updates to many programs, including OpenOffice.org. Appupdater is similar to apt-get or yum, but for Windows.
  • Integration with Bouncer and a Firefox extension like DownThemAll. (Both Bouncer and DTA have been patched to support Metalink, DTA patches have been merged but not released and Bouncer patches have not been accepted upstream yet).

Press descriptions

"Downloading bliss with Metalink" by Mayank Sharma:

"Getting popular software off the Internet can sometimes be a struggle, even with all the mirrors and BitTorrent Samaritans out there. When the Fedora project released Fedora Core 6 last month, for instance, even several dozen mirrors weren't enough to serve everyone, and torrent speeds weren't good enough because of a scarcity of seeders. But thanks to Metalink I was able to sleep while my FC6 ISOs were downloading.

Metalink is an open standard that claims to make downloading easier, faster, and more reliable by helping users extract the last drop of juice out of their connection. But Metalink isn't your run-of-the-mill download accelerator. It is in fact a framework for use by other download clients, and bundles traditional HTTP and FTP methods of downloading files along with BitTorrent.

The Metalink standard replaces static URLs with a .metalink file, which is a simple XML file. The file in turn contains locations of all the mirrors of the application you want to download. In addition to HTTP and FTP mirrors and rsync, Metalink supports several P2P methods as well, including BitTorrent, ed2k, and magnet links. For example, the OpenOffice.org metalink contains links to more than 50 HTTP and FTP servers and a torrent.

If I were using a regular hyperlink to download OpenOffice.org and the server went down midway through the transfer, I would be left with an incomplete transfer that could either be resumed or not depending on the download client. But with Metalink, if one server goes down, the client software simply jumps to another mirror and resumes downloading from that point. In effect, this increases reliability, as all listed servers would need to be down for the file to be unavailable.

Files downloaded with metalinks are automatically verified. Metalink supports both MD5SUM and SHA1SUM checksums as well as PGP signatures, which are embedded in the .metalink file itself. Checksums are unique for every file, like a fingerprint. If there's an error in transfer, or if someone has maliciously replaced a good file with a bad one, the checksums won't match. If that happened with another file transfer method, you'd have to download the file again from another mirror. With Metalink, if the application you are downloading has a torrent, Metalink can use a torrent's partial file or chunk checksums to verify mirror downloads as well. If only a small chunk of a download has errors, Metalink just re-downloads that part instead of the whole file."

OSresources article:

"Metalink is a system which improves the download process by increasing availability and guaranteeing integrity. It can give your users a more reliable download by providing multiple links to the same file, which can be switched to if one server is down or fails during transmission. It can also make downloads faster by using multiple resources at once. Organizations use mirror servers in multiple locations to provide redundancy and place content nearby users in the hopes that it will be faster for local users. Examples would be mirrors on each continent or near a concentration of users. Downloaders in Europe would manually choose a mirror located in a European country, preferably their own, and people in Japan or other countries would choose mirrors close to them. If mirrors are down they can be used by downloaders in other locations as a backup or second option. Metalink lists mirrors with machine readable information on priority and location so their efficient use can be automated by download programs. It can list mirrors around the world, but will automatically default to mirrors closer to you and by priority.

Download programs (also known as download managers) that are Metalink-aware read the information from a Metalink file (one with a .metalink extension), which is just plain XML. The download programs split the file tranfer into independently verifiable segments which are re-assembled. By downloading multiple segments from multiple servers, download speeds are usually three to four times faster on an average broadband connection and much faster on business grade connections. The checksum verification process, usually manual and arcane to most people, is automated with Metalink, so files are guaranteed to be an exact copy of the file you downloaded, free of errors. Metalinks can also contain publisher information, Operating System and architecture, language, file descriptions, mutliple files (to be added to a download queue) and so on. All this extra information allows download programs to do interesting things. It lets you have one download link for multiple Operating Systems and languages, and downloaders will get the correct version for their system."

TorrentFreak explains Metalink like this:

"We’ve all seen those download pages where a couple of http, ftp and bittorrent mirrors are listed. We often just pick one of these, but wouldn’t it be great if you could use them all at the same time? Metalink has the solution. It works like this, the .metalink file tells your download manager where it can get the file, and connects to these different sources at the same time."

downloadsquad describes it this way:

"Metalinks make complex download pages obsolete by replacing long lists of download mirrors and BitTorrent trackers with a single .metalink file. As you might have already guessed, a .metalink file is a file that tells a download manager all the different ways it can download a file. The file itself takes the form of an open XML standard that can list an unlimited number of HTTP and FTP sources as well as BitTorrent trackers and ed2k and magnet links."

External links

Personal tools