Finding the closest Apache Software Foundation mirror programatically

Question

For my deployment automation needs, I would like to dynamically and programatically determine the closest Apache Software Foundation mirror since the servers are distributed across geographically distinct locations and it would be ideal to dynamically determine the best mirror without having to hard-code that knowledge somewhere.

The only approach I could think of so far is to scrap the http://www.apache.org/dyn/closer.cgi page for the closest mirror suggested there, but it seems a bit cumbersome and fragile.

Is there a web API endpoint that provides this functionality in a stable and reliable way?

Answer 1

The mirror URLs in the page are marked up as <strong>, so you can scrape the page to get the top recommendation like this:

curl 'https://www.apache.org/dyn/closer.cgi' |
  grep -o '<strong>[^<]*</strong>' |
  sed 's/<[^>]*>//g' |
  head -1

Additionally, closer.cgi supports an ?as_json=1 query parameter to provide the same information as JSON. The result has a key of preferred for the closest mirror, as well as http for the alternatives.

Answer 2

There is a more elegant way by using jq:

curl -s 'https://www.apache.org/dyn/closer.cgi?as_json=1' | jq --raw-output '.preferred'

Answer 3

Here is an alternative using python:

curl -s 'https://www.apache.org/dyn/closer.cgi?as_json=1' \
| python -c "import sys, json; print json.load(sys.stdin)['preferred']"