Python 3, How To Use Python To Save Data From This Page?

Question

I am trying to save price data from this page using Python 3.x.

I want my script to go through every option under the Fund Provider dropdown, and then save the resulting table to a local file.

Unfortunately, when I look at the source code, it appears that all the menu options and table data come from JSON files, and I am not sure where to begin as I can't seem to read the files from a browser. I know how to use urlretrieve, and have used it for simple static web pages, but my skills are not advanced enough to navigate complex multiple resource documents.

Any advice on how I can achieve my goal would be most appreciated.

Answer 1

Sorry for doing an incorrect copy and paste with the URL. Anyway - I found a solution. What I needed to do is:

• use Firebug (an extention for Firebug) to identify the location of the json files, along with the posted information.
• then use urlretrieve to download the data, including post information with each request

example code:

from urllib.request import urlretrieve
import urllib

url   = 'http://www.example.com'
values = {'example_param1' : 'example value 1',
          'example_param2' : 'example value 2'}

data = urllib.parse.urlencode(values)
data = data.encode('utf-8') # data should be bytes

save_path = save_root + fund_provider + '.json'
urlretrieve(url, save_path, data=data )