how to find out latest file name from a website using a shell script

Question

I am relatively new to Shell Scripting and was facing the following issue. I want to download the latest freebase dump from http://commondatastorage.googleapis.com/freebase-public/ site. I know the file name format - freebase-rdf-.gz e.g freebase-rdf-2014-01-12-00-00.gz.

I checked and the "wget" command gets the file. However my problem is that I need to find out the latest data dump and then run the wget command to download the file from the site. I checked that the wget command doesn't accept regular expressions.

Can anyone help me with this problem ?

Answer 1

The latest version seems to be in the file named freebase-rdf-latest.gz. To get it with wget, use:

wget http://commondatastorage.googleapis.com/freebase-public/rdf/freebase-rdf-latest.gz

Answer 2

You can use the timestamp function of wget :

http://www.gnu.org/software/wget/manual/html_node/Time_002dStamping.html#Time_002dStamping

Store your latest download and wget cares to get the newer once.

Or you can get a directory listig (http://www.editcorp.com/Personal/Lars_Appel/wget/v1/wget_7.html)

If you specify a directory, Wget will retrieve the directory listing, parse it and convert it to HTML. Try:
wget ftp://prep.ai.mit.edu/pub/gnu/
lynx index.html

Parse it localy, sort it and take the last result for a wget call.