All geek questions in one place

Downloading a file from the correct Apache Mirror using wget

To use a specific example, I want to download the binary version of Hadoop 2.7.2. The website points to http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz , which then points to the nearest mirror by location. For me, this is http://xenia.sote.hu/ftp/mirrors/www.apache.org/hadoop/core/hadoop-2.7.2/hadoop-2.7.2.tar.gz .

I want to load this file into a shell script (specific Docker file). I would prefer to use the agnostic location URL to load so that if someone runs a script on the other side of the world, they will not use the same mirror.

Is there a URL that I could use with wgetor curlthat dynamically redirects to the nearest mirror? What will this URL be for this particular file?

+4
source share
1 answer

source closer.lua actually states that the query parameters actionand filenamecan be used to create a redirection to the requested file is automatically selected in the mirror instead of the usual mirror selection page HTML.

So, you can download files directly through this URL: https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=hadoop/common/hadoop-2.7.2/hadoop-2.7.2 .tar.gz :

GET /dyn/mirrors/mirrors.cgi?action=download&filename=hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: www.apache.org



HTTP/1.1 302 Found
Cache-Control: max-age=3600
Connection: Keep-Alive
Content-Length: 0
Date: Mon, 13 Mar 2017 18:08:00 GMT
Expires: Mon, 13 Mar 2017 19:08:00 GMT
Keep-Alive: timeout=30, max=100
Location: http://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
Server: Apache/2.4.7 (Ubuntu)
+4
source

Source: https://habr.com/ru/post/1649170/

More articles:


All Articles