How to download file after form submit and 302 redirect using wget or curl

Question

I'm trying to download a CSV file from the command line. You input a stock ticket into the form field, click download, and the file is downloaded. Here's the flow:

Site in question is here.

The flow goes like this:

1. Browse to http://www.cboe.com/DelayedQuote/QuoteTableDownload.aspx
2. Enter ticker into form field and click download
3. Form action POSTs to http://www.cboe.com/DelayedQuote/QuoteTableDownload.aspx
4. Response is 302 with location header set to http://www.cboe.com/DelayedQuote/QuoteData.dat
5. Browser follows redirect and GETs http://www.cboe.com/DelayedQuote/QuoteData.dat
6. Browser downloads QuoteData.dat which is the CSV I want

Pics showing request/response:

Chrome's RestClient shows the flow as well

Wget follows the initial 302 to the QuoteData.dat URL, but doesn't download that file before redirecting again.

I've copied the headers and cookies to use with wget and curl with no success. I've 'Copied as Curl' from the Chrome Inspector with no success. I've played around with casperjs as well, but I've gotten further with curl and wget.

I'm mildly obsessed with figuring this out now :) Any help would be appreciated.

UPDATE: Here is the wget command I'm using:

wget --post-file=cboe_form_data.txt -L http://www.cboe.com/DelayedQuote/QuoteTableDownload.aspx

cboe_form_data.txt is the post data taken from the chrome inspector: http://sandalsoft.com/cboe_form_data.txt

Answer 1

The site seems to require the Referer header:

wget --post-file=cboe_form_data.txt \
--header='Referer: http://www.cboe.com/DelayedQuote/QuoteTableDownload.aspx' \
http://www.cboe.com/DelayedQuote/QuoteTableDownload.aspx

With this command the "QuoteData.dat" GET request will feature the Referer header. Response code for that request is 200 and the CSV is included.

When the Referer header isn't present the "QuoteData.dat" GET request returns code 302 and an "Object moved" HTML page. In that case client is sent back to "QuoteTableDownload.aspx".

Answer 2

When you execute wget ....../QuotrTableDownload.aspx, it's download this aspx file,for the ...../Quitedata.dat ,wget treats them as foreign (they are not in ..../QuoteTableDownload.aspx), so you need the --span-hosts switch. Going to ...../QuoteData.dat , this option might help.:)