Retrieving HTML form submission in Python when there is a loading page

Question

I'm trying to automate a search on the conserved domain database (CDD) - http://www.ncbi.nlm.nih.gov/Structure/cdd/wrpsb.cgi with python where I submit content to a HTML form. The problem is that once the form is submitted, a new page loads saying to wait while the search is completed before after a minute refreshing the page with the results. All of this occurs at the same URL and when I automate the form submission all I can retrieve is the loading page as the result of the submission - not the final results page. I've tried to use sleep etc to delay but the form submission gets the first page that loads back no matter what. I'm using the Python requests module but I also tried Mechanize without much success beyond retrieving the content of the loading page.

import requests

url = "http://www.ncbi.nlm.nih.gov/Structure/cdd/wrpsb.cgi"
payload = {'seqinput':'AATCACTTCAGCACACGAGCTTATATGCTGCTTTATTGGAGAACCGAGTTGTCCTTTATTGTGGATGCAATCCCTTGTATGTGTTATACTTTATGCTAAAGTCATCGACCTTTGCGTAATCGATCGTAAATCCTGGTATAGGGTGTACTTCGGATTGCCCGAGCCTATATTGGGGGTGAACGGATCATAAAGTCACTACCATATTAAATTGTACTAAGTCACGCCCTGAATATCTCAGGTTTTGATTTTATCACATTTTATTTTAAACCCTAGTTAAAGCTTTACAACTTTAAAGTAAGCCGTATCGAAACGTTAATCGATCGCTGATTGCGTAACAATCAATAGTTAGTTTTATTTTTAATTTTCAATATTTATTTTTCAAAAATTTTAGTTAAGATTTTAGCTTGCCTTAAGCAGTCTTTATATCTTCTGTATTTAGTTTTAAAATTTTTAGGAGCTAAGTTCGCTTTGCTCTTTATAGCTATTTTATTTATTTTAGGAATTTTATCACCTCGTAATTATTTATTAATAACGTTAGCTCTATCCGTATACAATGTCTACGATACGTTGCACCAGTTTACCAAAATCGGTGTTGCGCATTCAACAACATTGCAGTAGTTTAGCTGCTGAACCTAGGCTAGTAACATTACCGAATAAAAAGTTTAAAGATTTCGTTAAATCATTTGAACACCTTGATGTGCCTCGACCATTTAATGTTTTAATGGAGTGGTCAGAGGTGTGTGATAATGATTACACGTTAGGATATTGCAAATCTTCTACATTATATGAACAAATGTTTATTTTGAAATATATGTTGGGTTTTAATTTTGAACCGCGTAAGTTGATCATACTCAAAAGACGAATGCAAGAGCGTTTGTTTTTTTTTTATAGAGGTTCCCGGACTATTATACCTTTGAATATGGTTTACCCATTATTTAAAGATACATTGAATAGTATCGGGCATCCAGATCATTTAAAACGACACGGATTCGAATATTTAGACCAAACAGATATACAGAAATTTCCATTTTTAATATTGATGTTTTGTATTAACATTGAACGTGTACCACTTAAGTGTTACAAATACAGTGCTAATATAAACGTTAAGGTTGCTTTGTCTCAATTACCTGTTTTAATGACTGCCCTTGTTAACAAGGAACGGTTAGTAGATCAGTACGATTTTCTAGTAGTATTACAAAATTTTCTACATTTTTTTTTTAGGAATTTAAGTTATACTAATCCAGTAGATTTAATTGAACATAGTTTTCTTAACGATTTAGGTTTGTACTGCGCGAAATTAGTTATGGTCAGACCCATATCTACCACTGAGTACCTACTTGGTGGCGATTGGGATTTGGACAGACCTTGTTTAGCGCCAAAGGAAACATGGGATGTATATTATCGGCTGCAGTCTGTTGCACAATTAGGTGTTGATGTCAACACTAATGTGCAAATGGATCCTGCATTTATACAAGCCTTGGACAAAATTGCCGAAACATTTGGAGAAGCGGTTAACAGCTTGCATTCAGTTGCTGGCGTTGCCGTGGATGGAATCAGGAAGAAATTTGCATGTTACCTATCTATTTGTTATAATTTGTATAGACTAGGTTCTGGTGGTATGTCTCCTCAAGACGTACTTATGAACATAGTTACTACACTTATGCAATCAGATATGCCTGCAAATATTATTCCACAATTGCGAAGTATATTTTTAACATCTACAGCGCAGTCTGCTACTGTTGATGGAATGGTAATTGCAAAGTTATTAGCTTTGTGTTCCTTCTCATTGATGGTCAGCAAGATACCTTCATCCAGAGACATTGATTCATTCATATTACGTCTAGACAGAATACCGCGAGCGTTTTCTGGTTTAGAGAATATGTGGAAACGGTTGGATACCGTGACCAATGAATTGTGGACTTGGATGGAGATTACTGTGTTGAAGCGTGAGAATGTAATTCCGCGTTCTGATATCCTTGATTCTGTTAGTAAGTGGGAAAATGACCTTGAACAGTTGTTGACTCTTCAGAAGCATCGAGAGATTCAATCGAGTCTCGAGACGCAACATGCCGCAGGGAGAATGTATTCCGAAGGGATACGTCTTATGCGTGTGTGTAAAGATCTGAACCTTTCTAAGGGAAACACCGAAATCATCGCCCGAAATTTACCAGCCGCAAAACTTCTGTTAACGGAAGCTAACATGAGTGGTGCGGACAAATCAAAATTACGAACTGAACCTGTCATTGTTTGGTTTTCCGGAGCTTCTGGCAACGGAAAAACTGGATTATCTTACCCGTTCATATTAGATATGATGCGAGTATATGGTGATCCACCGAGCACGTGGCAGCAAAATGTGTATGCAAGAGAGCCTGAGACAGAATATTGGGATGGTTATATAAATCAAGAGTATATTGTCTATGATGACTTTATCCAGATTAAGGATTCACAATTGAAACCTAATCCGGAGTTGTTTGAGATGATACGCTTAGGAAATATGTTTCCGTACCAATGTCATATGGCTTCCTTGCTTGACAAGAACAATACTTTTGCCGAACCGAAATTAATATGTTTAACTTCTAACTTACAACGTTTACAAATTGAATCACTAAACTGCCCCGAAGCTGTATCCCGTCGTATAGATTTTGCTTTTAACGTTAGAATTATTCCAGAATATCAAATGGAATATACTAGCGCTAATGGCGACAAGCTATATAGATTGGATGCTGCGAAGGCAAGACGTGATTTTGGAGACGTTCTTTGTTTTGAAGTATATAGATTTGACATGTTTGATGCCTCTAGTCGCCGTGATATTTTGACGGACCTAACTTACACCGAAATGGTTAAGTTATGTCAAGATAAAATGCGCGACAGAGCTTCGAATTTTACCGACTATGCCAATTTCTTAGAATCGTACAGGAATAAGGGGGTTGCTCAAGTTGAGAAACCCAAACATGAAACTGACGATTATAATGGAGAGACCATGATTTTTACATCTACTGCCCAAGTGCATTTAGAGGACATTACTAATCTAGTAGTGCCACAACCTAGCTATTTTAAGCGTTTATACTGGAATATGTGTAAACAATATTATAGTACAAAGTTGTGGCTTACTGGATCAGATACTAGTGTCTTTGAGATGTTACTTCTCGGTGATAGAGATGGCGCTTATGACAAGTGCCTCGCTATTGTTAGAGAAACTAGATGTGAATTGAACAACATGATTAATAGGGAAGCAGAAGTTATTAAAAGTGTCTTTGGGAATTATTGGCCACTCTTTAAAGCTTGTGCGGGTGCTGCAATTGGTGCGTTTTCGCTTTATTTTATTCTAAGGAAGAAAAGCACACCAGTTACGGCCTTTGTTGCGAGTAATAAAGAGCTAATGGAAACTATGAAGAAAGCAAATGAGTGTTTAGATAATGAGTGTAAGCATTGTAAAAAATGCTTGCATAAAAATGTAGATCTGTGCGTGAAATGGTACACAAAATGCCATTGCTACGCGCTACAGATGGAATCTGCACAAATTAATCTTAAATACTACGCTGCTGCTGCTATGTATCAAGAACCCGAAATGAAAAAAGAACGCGACCGATGCGTAGAATTATTATCAATAATTGATCAATTGTGTAGTTGTGACTGCGCAAATTGTGATGCTTGTTGTGATGATTCTCTCGCTGAAAAGTTTGAAAATGTCATCAAAGTTTATGAGATGCCATGTGTCTGTGTGTGTGCTCGTCTATCACAAGGATTTGATATGATTGAGTTACTAGCACTAATAAAACATTGTGGCACCTTAGAACCAACACCAATTTTGAACCCATATTTACGTAAACTAAGCGTTAAACTATCCGCAGACGTAAGAGATTTCGAAAGAACAGCAGATTATGAACAACTGCTGAACACTCTTCAATCTCAGGAATACGAAGGAGACGTTAAACCGCAGACGATACGTAAAGTAGCAATACGTTACCAATCTCATGATGATGATACCAACATGCGATTGCGTAAAGTGTTGCCTCGCGTTAAATACCAAACAGAAATTACTACAGACGAAACCCGTAGCGTGAGCTCCGCGAAAGAGAGCACCCATATCGACAAAGTGACTAACAATATTATGATAGACGAACAACGAGCTATGCCAGAGATGGATAAATCCGTTGAAACTATAGTAAACCATGTTGTATATCCTAATACAGTATATATGACAGCCAACAAGAATGACGGCAAACAAGCCAACATTGGACACATAATATTTGTGTGCGGCCAAGTTGCCTTGATGCCCTATCATTATAAAGTTGCTATAGAGGAACGAAATTATTCGTCCGTGAATTTATATTCACGTCAATTGATTGGTTCTAAAATACCTGTCTCTGTGTTTGATACATTCGTACGCATCCAAGGAAAAGATGCCATGCTTGTAGCATTCCCAGTTACAGTTAATAGTTTCAAAAATATTGTTAATCATTTTGTGGATATTCAAAACTACCCCCTAGTGCCGTCATGTCCCGGCATACTCGCTAAGTACTATTTTGCTAACTCTGAGACGGAGAAATCTAGAGTTTGCATTAGTGCTATCGGCGTGTCGGAACGTGATGAAGTGGACGTCATGTCTGTCCCTGGATGTATGGAGGTGGTACGCAATAGAGATTTCTACACATACACCGCGCCAACCCGCGCTGGTGATTGTGGTGCAGCTCTTTGCGTTGCCAACACGTGCATACAGGGAAAGATAGTCGGAATACATGTATCCGGCGTAGAAGGGCTATGTAAAGGCAATTCTTCCGCGATAACCAAGCAAATGATAGAAGAATCATTGAAGAAAATGCCGAGCATTGCTCAATACGCATACCCATCCTCTGAACTAACCGTTGAAATGGACGTGTTAGAAGAGAGTGGAGCATTTGTATTACACAAATATTTGCCAGGGGTTTCTATAGGTACAACCATGCAGACTGCCATTAAACGCTCTCCAATTCATGGCGAACTTATAGAATCTCCGAACAAACCAGGACCGCTTGGACCCTTTAAATTTAGAGGGACCATGGTCGATCCGCGTGTGTTACAACGGAAAAAATATGGAAAACCACGTCCGGTTATCAACCAACAAATAGTAGATGATATTAGAGACGGTTTAAAACCTATTTATTATCAATCCCACGAATACGAACCTGAGTACTACAAATATCCATTAACTTTTGATCAAGCAATATTGGGTATAGATGGTGATCCGTTCATTAATTCACTGGATCGTAATACAGCACCTGGCTTCCCCTTTTCTACACGGAGAAATGGAAAGAAAGGAAAAACGTTGTGGTTTGGAGACAGTATGGAATACGACCTTACTGGACCACACGCGATGGCATTACGACAGGAAGTTGAAGAATTGGAATTATCTATACTCAATGGTGTTAGACCTGAAGTTGTATGGACCGATACTTTAAAAGACCAGAAAATACCTGTAGCTAAAGCAAACGCTGGTAAAACACGTTTGTTTTCAGCAGCGCCAATGCATTATGCAATAGCTCTGCGGAAAGTGTGTGCCCCTTTTGTTGCTCATCTATCACGGATGCGTATTAGAAATACGATCTGTGTGGGTGTGAATCCGTTCTCATGCGAATGGAGTGCGATAGCACAAAAATTGTCATCGAAGGGACACCATGTTATAGCTGGAGATTATTCTAATTTTGACGGTTCATTACCTGCTCAACTAGTCTACGCGGCAACTGAAATAATGGCAGACTGGTATGACGTTCATTGGGAATACGTTGAAGCTCACAAGCGTAATATCGTTGGAGGCAATGTATTAGGAAAGCCAGAATTTCTTATGTACTTGCGCCGATTATATTATGAATGTGTACATCATTTACATATTATGAATTTTAAACAAGGTTCGCTTATGTATTATGTTCGCAACGGTATACCCTCTGGATGTCCAGTTACTGCGCCATTAAATTCAATTGTCAACTTAATGGCATTAATCTATTGTTGGTATCATATAATAGATGATCCACTCAAGCAAAATGTTAAAGAGTTTTTTGAACACACCTCGAGTGTTTTTTATGGAGACGACTTCGTAATGAACATCCGAGCAGATGTATTGGAGAGATTCAATCAGATAACGATAACACAAGCTATGAGTGAACATTTAGATATGACTATGACAGATGAAGCCAAAACGGGTGAGTGTATTAAATCTAGGACACTAAAGGAAGTTAACTTTCTCAAACGTGCTTTTTATTATAATACGCTTATCCAAGAGTACACCGCACCGTTGGACCTCACGGTTATTTTAGATTCTACGAATTGGTATAAAATCGGTAAAAGTTCTGCTATAATAGTGGCGCGTGATACGCTCAAAGCGTGTTTACGTGAATTAGCTTTACACCCGGAACATGTCGATTTACAATATCGTAATAAAATAACAGATCTAGGTCTTCGCGTTACTAATTTAATTCCAGGAGAGTTATTTGTGCCTGATACAAGGTACTCGACGCTTCTTGCTATTAAGAATATGGAATGTGAAAATTTAGGTTTGGACTGTGACACTTAAGATAGTCTAATTAGAAAACCACGTAATTGGGTCAATCCGTTAATACCGGTCTACCAAGCCCTGGAAATTTCTAGTAATCTAATAAGTGCAACACCGCATCTACTCTAAAATCGATATTTGATATCATTAGTCTAATTAACAAACCACGCAACTATGTCAATCCGTTAATACCGGTCCAATAGGCCCTGGAAATTAGTAGTAATTATTATTATACGTCAACCCTGCTCATTGGTTTAATTGAGCACTTATGTTGCTATGTGATCTTGCATAATAAATGCTGACGTGAAAACGTTATGCACTGCTGTAGCAATAGGTTAGCTATTTAGCTTTACTAATCAAGACGCCATCGTGCAGCCCACAAAAGTCTAGATACGTCACAGAAGCACATACGCTAGGTCGCGTTGATGCTTCTCATACATGACCTGCAAATATAAATGAAAACGTAACAACCAAAACACAACAACAAATCCTATCTTTCAGCTCCGAAGGTGAAGCCCCGTCGTCGTCTACCGTTTTAGCCCCGCTCAAATTGCAAAATCCAATTCTGGATTGTGCAAGGGATGGAAAGACTCATACAGTCAATTCATTCCTTGAACGGCCTATTAACTTTAGAACGGCTACGTGGAGTAATCAAGCTGTTGGAAGTAGGTTATTTTCTTTTAACTATCCTTCGGATGTGGTAAAGAATCCAATGTATAGTAGAAAATTACAAAATTTTCTTGGTTTGCGAGCTGATTTAGTTGTTCGCGTTCAAGTCAACGCACAACCATTTCATGCTGGTAGACTAATGCTATCATGGACCCCTTTTCTTAATTCGTTGGGTGCTAGTAGAAAATATTACTATACCAATCCTACATCGCCGTTTTTAACCTGTATTAGTGGTAATCCTCGTGTTGAAATAGATCTATCTACTACAACTGAAGCTACCATGACTATACCTTTCGTGTCTCCTTTCCTATACTATAATCTGGTGACGGGAACTGGCGATATTGGAACTTTCCAATTAATCGTCTATTCTCCGTTAGTAGATTTGGTATCTGGAGGGAACATAGACTACACAGTATGGGTTAATATGACTAATGTACGTACCGAATTTCCTACTGGTATGCCAACTTCTACCGCGCAAGTGGGGGAAGAGGGTAAACAACAACAGAAGCAAGGTTTTGTCACACGCCAGGCCGAGGCTTATTCTACTATTATGGAACCACTCACTAAGATACCAGGAGTAGGCCAATTAATAGGATATGCCAAGTCGGGCGTTGACGCTTTGCATGCTGTTGCTGCGACCCACGGTTGGTCAAAACCACTTAATCCTGCGGACATGCAATTATTCAAGCAAGCACCATCTCGGTTTATGTGTAATTCTGATGGTTCGGATATGGCAACTAATTTGGGCTTAACAAGCCAAAATGAAATTGAGCATCTCCAATCGTTGTTTCGCACAGATTCTGATGAAATGTCTGTAGATTACGTGGCTAGAACATATAACTATGTAGGAAGATTTGACTGGAAGAAGGGAGATAGTCCAGGTACTATTCTTTATAATCAAGTTGTTTCACCTACAGCTTGGTTCTCTAGGATTGGTATTACTGGATTGTCTATTCCTCATTTATATTTTGCCGCTTCAAACTTCGTGTTATGGCGCGGTGGTATAAATGTAAAACTAAAATTCGTCAAAACAAAATTCCACTCAGGTCGTATACGTATTATATACGTACCCGGCTTTTTCGGTGGGGTTTTACCCGTTGGATTTGAAACTGACGCTAACTATTCTACTGTGGTAGATATTAGATCTGACACAGATGTAGAATTTAACGTCCCTTACGTGGCCACTGTGCCTTGGTTGCATATCAATTCCACTCCATGGGTTACTAATTTTAACCAAACTCATGCTTGTGGATCGATTGTAGTCGAGGTTCTCAATGAACTCGTAAATACATCTACTGTATCTGATACTATCGAAGTTATTGTTGAAGTGTGTGCTGCAGAGGATATCGAATTCGCTATCCCTATAGTACCCGCTTTAGCACTTCGTGCGCCACCTAATAACGCAAGTAATAAGGGTGTTTTGGACATTATTACAAGCATGGCACAGGTAGGAACAGATACAGGTGATACACCATCAGAGGTAGCTCGTGAAGAACCAACAACTTTTAACGAGGTGCCGTTACAACCAACTACGACAACATATAACGCATCTATGCTGATGATGGGTGAGAAAGTTACTAGCTTTAGACAGCTTATAAAGCGGTTTTCTGCTATAACACCACCCACGCAGAATAGATATTGGGAATTCAAACAACCTTTTTGGATTAATCCAAATAGGTTCGAAGGAATAACAAGTGAGGGTACGTATGATATCGACGGTATCTCATGGTTCGCAAGCTTATATGCATTCTATCGAGGTAGTATGAGATATAAGATCGCGCCTATCAGTAATACTTCACCCCTTGTTGTAGCCCTTAAACCCAACTCCTTATATGCCGGTATTCGAACTATCGACATAAACGGTACTTGGGAATATCCCGACTACAAGGGTGCAGAAGTATTTATGACACCGAATGAAGGAATACATGAGTTGAGCATTCCATACTATAGTTCCTATCCCGTAACTTTAACTACGTATAACACTAGTGGTTCTGACGTACTTGATGCTAGAAACGGTTTTAATCGTGTTATAGCTCGGTTTCATCAGGACACTAATGCTTACGTATATAGGGCGGCGGGAGATGACTTTAGTTTTGGATTCCTCCTCGGACCTCCTATAGTGAACCACGCATCCCCACAGCGTTCGTCTGGTACGCGTTAGTTACCATCAAGTTTATAGTCTTGTTAAAACTATACGTCTGATATGCGTTAATTATCAAGTGAGTAGCTATCTCGAATTAAAATAGCCGTCCAGTGGAACGTATCTACTTGACTTACAAACCAATACGAAGGATAAGAGTATTCGTCGCATTTGATTTTAGCGTCGCTAGGCCTAGTTTGTAAGTTAATCTAATTATTATTAAGGACCAAAAAAATATAAAATACAAAATACAAAAATATTTAGCATTAGTTAATTTTAGGAAATAATTTTAGATAAGCCTGGCGCGCCAAAATTGGAGTTTAAAGACCCTATAAAAATGCTGGCGTTATACACCCCTTGTTCCTTCGAGGCGGGCCCTCATGTGTTCCACATGCAGACTTAAAGAAGTCTACCCCTGGCCCGTCCGGGGGATCTAAGTTTTTATCAGTTTTCTTAGACTCAGCTATTAGCTGAGAGGCGTAACTGTTACTTTATGTCTGGCGAACATGAGAGGTGCTGTAATGTCATTTTTCGTCAAAAAAAAA'}

r = requests.post(url, payload)

print r.content

Is it possible to refresh the data you get back from a page submission in Python?

EDIT

Thanks so much for the advice sberry, I checked the loading page and found this hidden form. The first loading page waits for 3 seconds before refreshing, then the second loading page waits 8 seconds and the third one 13 seconds so it looks like I'm going to have to post and check the result a few times. Now I just have to figure out how to do this.

<div id="display">Please wait while your data is being prepared...<br>RID = VGCR81NG014</div>
<div id="counter">Page is refreshing in <span id="seconds">8</span> seconds</div>
</div>
<form id=_refresh action=http://www.ncbi.nlm.nih.gov/Structure/cdd/wrpsb.cgi method=POST target="_self">    <!-- the form to submit for refresh or redirection -->
<input name="dhandle" type="hidden" value="QM3-data_cache-22B43E7AD59D887C-51B832F442D7"><input name="output" type="hidden" value="html"><input name="wait4blast" type="hidden" value="10"><input name="mode" type="hidden" value="rep"><input name="data" type="hidden" value="ftable"><input name="gwidth" type="hidden" value="-1"><input name="loading" type="hidden" value="true">
</form>

Answer 1

import time
import webbrowser

import requests
import lxml.html


session = requests.session()
url = "http://www.ncbi.nlm.nih.gov/Structure/cdd/wrpsb.cgi"
form_data = {'seqinput': '......'}

try_count = 0
while True:
    try_count += 1

    print 'Request....'
    r = session.post(url, form_data)
    with open('{}.html'.format(try_count), 'wb') as f:
        f.write(r.content)

    root = lxml.html.fromstring(r.content)

    form_data = {input_.get('name'): input_.get('value') for input_ in root.cssselect('#_refresh input')}
    if not form_data:
        break
    form_data['tick'] = '10000'

    print 'Wait for 10 seconds.'
    time.sleep(10)


print 'Done'
rid = root.cssselect('#div_search_info table.searchdata tr:nth-child(1) td:nth-child(2) strong')[0].tail.strip()
webbrowser.open('http://www.ncbi.nlm.nih.gov/Structure/cdd/wrpsb.cgi?RID=' + rid)