Posting form says that I didn't provide input

StackOverflow https://stackoverflow.com/questions/23138639

  •  05-07-2023
  •  | 
  •  

Question

I'm trying to post to a form on this page: http://stitch.embl.de/cgi/show_network_section.pl?identifier=802634%20802659&additional_network_nodes=0&chemicalmode=10.5&input_query_species=7955&interactive=yes&internal_call=1&limit=10&minprotchem=1&network_flavor=actions&required_score=400&sessionId=0ZKGxMnojGDg&targetmode=proteins&userId=Ri1Zp7hT9L9G. The form that I want to post to is (Info & Parameters ... at the bottom of the page):

<form action="/cgi/show_network_section.pl" id='standard_parameters' method='post'>
  <table border='0' cellpadding='2' cellspacing='1'>
    <tr>
      <td colspan='3'>Network Display -  Nodes are either colored (if they are directly linked to the input - as in the table) or white (nodes of a higher iteration/depth). Edges, i.e. predicted functional links, consist of up to eight lines: one color for each type of evidence. Hover or click to reveal more information about the node/edge. </td>
    </tr>
    <tr><td colspan='3'><hr/></td></tr>
    <tr>
      <td colspan='3'>Active Prediction Methods:</td>
    </tr>
    <tr>
      <td colspan='3'>
      <input name='channel1' type='checkbox' />Neighborhood       <input name='channel2' type='checkbox' />Gene Fusion       <input name='channel3' type='checkbox' />Co-occurrence <br/>      <input name='channel4' type='checkbox' />Co-expression      <input name='channel5' type='checkbox' checked='checked'/>Experiments       <input name='channel6' type='checkbox' checked='checked'/>Databases       <input name='channel7' type='checkbox' checked='checked'/>Textmining       <input name='channel8' type='checkbox' />Predictions       </td>
    </tr>
    <tr><td colspan='3'>&nbsp;</td></tr>
    <tr>
      <td style="white-space:nowrap">required confidence (score):</td>
      <td align='left'><img src="http://stitch.embl.de/images/small_white.png" width='70' height='5' alt=''/></td>
      <td style="white-space:nowrap">interactors shown:</td>
    </tr>
    <tr>
      <td style="white-space:nowrap">
        <select name='required_score'>
  <option  value='900'>highest confidence (0.900)</option>
  <option  value='700'>high confidence (0.700)</option>
  <option selected='selected' value='400'>medium confidence (0.400)</option>
  <option  value='150'>low confidence (0.150)</option>
        </select>
      </td>
      <td align='left'><img src="http://stitch.embl.de/images/small_white.png" width='70' height='5' alt=''/></td>
      <td style="white-space:nowrap">
         <select name='limit'>
   <option  value='5'>no more than 5 interactors</option>
   <option selected='selected' value='10'>no more than 10 interactors</option>
   <option  value='20'>no more than 20 interactors</option>
   <option  value='50'>no more than 50 interactors</option>
         </select>
       </td>
    </tr>
    <tr>
      <td align='right' style="white-space:nowrap">or custom value:         <input name='custom_score' size='8' maxlength='8' /></td>
      <td align='left'><img src="http://stitch.embl.de/images/small_white.png" width='70' height='5' alt=''/></td>
      <td align='right' style="white-space:nowrap">or custom limit:         <input name='custom_limit' size='8' maxlength='8' /></td>
    </tr>
    <tr>
      <td colspan='3'><hr/></td>
    </tr>
<tr><td colspan='3'>
<table border='0'>
<tr>
<td style="white-space:nowrap">additional (white) nodes</td>
</tr>
<tr>
<td style="white-space:nowrap; text-align=center;">
<input name='additional_network_nodes' onchange="UpdateNavigationButtonsFromForm (this); return true;" size='10' value='0'/>
</td></tr>
</table>
</td></tr>
<tr><td colspan='3'><hr/></td></tr>
    <tr>
      <td align='left' style="white-space:nowrap">        <input type='submit' value="Update Parameters"/></td>
      <td>
        <input name='identifier' type='hidden' value="802634 802659"/>
        <input name='required_score' type='hidden' value='400'/>
        <input name='limit' type='hidden' value='10'/>
        <input name='UserId' type='hidden' value='Ri1Zp7hT9L9G'/>
        <input name='sessionId' type='hidden' value='0ZKGxMnojGDg'/>
        <input name='network_flavor' type='hidden' value='actions'/>
        <input name='external_payload_URL' type='hidden' value='_unassigned'/>
        <input name='internal_payload_id' type='hidden' value='_unassigned'/>
        <input name='chemicalmode' type='hidden' value='10.5'/>
        <input name='input_query_species' type='hidden' value='7955'/>
        <input name='internal_call' type='hidden' value='1'/>
      </td>
      <td align='right' style='white-space:nowrap; black'>      </td>
    </tr>
  </table>
</form>

with the following Python (Python 3) script:

# python imports. Some might have to be installed first
import urllib
import requests

# Change to the URL that you get after inputting your genes. This is before filtering for 400
stitch_results_url = 'http://stitch.embl.de/cgi/show_network_section.pl?identifier=802634%20802659&additional_network_nodes=0&chemicalmode=10.5&input_query_species=7955&interactive=yes&internal_call=1&limit=10&minprotchem=1&network_flavor=actions&required_score=400&sessionId=0ZKGxMnojGDg&targetmode=proteins&userId=Ri1Zp7hT9L9G'
# the custom limit of the interaction shown
interaction_shown_custom_limit = 400
# parameters for active prediction methods
neighborhood = 'off'
gene_fusion = 'off'
co_occurrence = 'off'
co_expression = 'off'
experiments = 'on'
databases = 'on'
textmining = 'on'
# required confidence score
required_confidence = 'high confidence (700)'

# Input parameters we are going to send
parameters = urllib.parse.urlencode({
  'channel1': neighborhood,
  'channel2': gene_fusion,
  'channel3': co_occurrence,
  'channel4': co_expression,
  'channel5': experiments,
  'channel6': databases,
  'channel7': textmining,
  'required_score':required_confidence,
  'custom_limit':interaction_shown_custom_limit
  })

parameters_payload = bytes(parameters, 'ascii')
# Build our Request object
req = requests.post(stitch_results_url, parameters_payload)
print(req.text)

print(req.text) gives me the html that I pasted at the bottom of the page. It says that no input was given. Why does it not recognize my data input?

HTML:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-type" content="text/html;charset=UTF-8" />
<link rel='SHORTCUT ICON' href='http://stitch.embl.de/images/favicon.ico' />
<title>STITCH: chemical association networks</title>
<script type="text/javascript">
<!--
function loadFPWindow (node, label) {
   displayFloatingDiv('fpWindowDiv', label,-1,-1,mousePositionX-350,mousePositionY-10,'#DFDFDF',null, 'http://stitch.embl.de/cgi/show_item_info.pl?UserId=xrrQRRQ9bQjQ&amp;sessionId=k8xp6Dr4j42R&amp;node='+ node + '&amp;taskId=CufHLCVJEi2x&autosize=1' + '&amp;targetmode=proteins&noAction=1&search_string_link=1', 'http://stitch.embl.de/images/loading_bigrotation.gif');
   return false;}

function restoreFNButtonMode(event, handler){ 
    var flashNetwork = document.getElementById('StringNetworkFlex'); 
     if(flashNetwork != null && isMouseLeaveOrEnter(event, handler)){ flashNetwork.restoreButtonMode();
}} 
-->
</script>
<script type='text/javascript' src='http://stitch.embl.de/javascript/newstring_scripts_7018.js'></script>
<script type='text/javascript' src='http://stitch.embl.de/javascript/jquery/jquery-1.7.1.min.js'></script><script type='text/javascript' src='http://stitch.embl.de/javascript/basic/basic_js_util_7018.js'></script>
<script type='text/javascript' src='http://stitch.embl.de/javascript/basic/basic_ajax_7018.js'></script>
<script type='text/javascript' src='http://stitch.embl.de/javascript/basic/basic_floating_div_7018.js'></script>
<script type='text/javascript' src='http://stitch.embl.de/javascript/ac_oetags.js'></script>
    <script type="text/javascript">
        <!--
        function doneLoading() {
            var img = new Image();
            img.src = 'http://stitch.embl.de/images/smallest_logo_p.png';
        }
        //-->
    </script>

<link href='http://stitch.embl.de/css/floatingDiv.css' rel='stylesheet' type='text/css'/>
<link href='http://stitch.embl.de/css/standard_styles_pc.v7266.css' rel='stylesheet' type='text/css'/>
</head>

<body id="stringBodyTag" onclick="hideFDifInside('fpWindowDiv');" onmousedown="hideFDifInside('fpWindowDiv');">
<div style='position:absolute;' class='nodeInfoDivStyle' id='fpWindowDiv' onmouseout='signalFDOnMouseOut(event, this, "fpWindowDiv"); restoreFNButtonMode(event, this); window.status=""; return true;'></div>
<script type='text/javascript'> $("#fpWindowDiv").mouseenter(function() {signalFDOnMouseIn("fpWindowDiv"); });</script><div style="width:100%;text-align:center">
<table style='padding:10px;display:inline-table;' border='0' cellspacing='0' cellpadding='0'><tr><td>
<table border='0' cellspacing='0' cellpadding='0' style='visibility:visible;min-width:965px;'>
<tr><td class='img_compact' align='left' valign='middle' style='height:42px; width:42px; white-space:nowrap;' bgcolor='#4060FF'><img src="http://stitch.embl.de/images/small_fillpic_emph_pc.png" width=10 height=5 alt=""><a href='http://stitch.embl.de/cgi/show_input_page.pl?UserId=xrrQRRQ9bQjQ&amp;sessionId=k8xp6Dr4j42R'><img src='http://stitch.embl.de/images/logo_still_pc.png' width='36' height='36' alt=''/></a></td><td class='img_compact' align='left' valign='middle' style='height:42px; white-space:nowrap;' bgcolor='#4060FF'><a href='http://stitch.embl.de/cgi/show_input_page.pl?UserId=xrrQRRQ9bQjQ&amp;sessionId=k8xp6Dr4j42R' style='text-decoration: none'><span class='top_menu_link_c' style='font-size: 31px;'>&nbsp;STITCH&nbsp;</span><span class='top_menu_link_c' style='font-size: 31px; font-family: FolksLightRegular, Helvetica, sans-serif;'>4.0&nbsp;</span></a></td>
<td class='img_compact' style='text-align:center' style='height:52px; white-space:nowrap;' bgcolor='#4060FF'></td>
<td align='right' valign='top' bgcolor='#4060FF'><div style='margin:0.5em; white-space: nowrap'><a href='http://stitch.embl.de/cgi/show_input_page.pl?UserId=xrrQRRQ9bQjQ&amp;sessionId=k8xp6Dr4j42R' class='whitenondecorated'>Input Page</a> | <a href='http://stitch.embl.de/cgi/show_download_page.pl?UserId=xrrQRRQ9bQjQ&amp;sessionId=k8xp6Dr4j42R' class='whitenondecorated'>Downloads</a> | <a href='http://stitch.embl.de/cgi/show_info_page.pl?UserId=xrrQRRQ9bQjQ&amp;sessionId=k8xp6Dr4j42R' class='whitenondecorated'>Help/Info</a> | <a href='http://stitch.embl.de/cgi/show_my_page.pl?UserId=xrrQRRQ9bQjQ&amp;sessionId=k8xp6Dr4j42R' class='whitenondecorated'>My Data</a>&nbsp;</p><p><form name='navform' id='input_form' action='http://stitch.embl.de/cgi/show_network_section.pl' method='post' enctype='multipart/form-data'><input type='hidden' name='required_score' value='400'/>
<input type='hidden' name='limit' value='10'/>
<input type='hidden' name='UserId' value='xrrQRRQ9bQjQ'/>
<input type='hidden' name='sessionId' value='k8xp6Dr4j42R'/>
<input type='hidden' name='input_query_species' value='auto_detect'/>
<input type=hidden name='have_user_input' value='2'><input name='identifier' id='single_identifier' value='search' onfocus="onFocusCheckEntry(this, 'search', true);" onblur="onBlurCheckEntry(this, 'search', true);" style='color:grey'/> <script type='text/javascript'>
<!--
document.write ('<a href="#" onclick="document.navform.submit()" class="whitenondecorated">GO!</a>');
-->
</script>
<noscript>
<div><input type='submit' value='GO !' class='stringSubmitButton' style='width: 54px; height: 25px; text-align: center;'/></div>
</noscript>
</form></p>
</div>
</td>
</tr>
<tr><td colspan='4' valign='top'><script type='text/javascript'>
<!--
//-->
</script>
<noscript>
<div style='color: red; font-style: italic;'>Warning: your browser appears to have JavaScript disabled.<br/>Some of the features in STITCH may not be available to you<br/></div>
</noscript>
<div style='width:50em;text-align:center;margin:3em;'>
<table style='display:inline-table;'><tr><td><table style='text-align:left;' cellspacing='0' cellpadding='0'>
<tr><td class='info_box_title'><div style="background:url('http://stitch.embl.de/images/infoheader.upperright_pc.png') no-repeat right top;">
<div style="background:url('http://stitch.embl.de/images/infoheader.lowerright_pc.png') no-repeat right bottom;">
<div style="background:url('http://stitch.embl.de/images/infoheader.upperleft_pc.png') no-repeat left top;">
<div style="background:url('http://stitch.embl.de/images/infoheader.lowerleft_pc.png') no-repeat left bottom; padding: 8px 12px 7px 12px;">
error ...</div></div></div></div></td></tr>
<tr><td class='info_box_body'>
<div style="background:url('http://stitch.embl.de/images/infobox.upperright_pc.png') no-repeat right top;">
<div style="background:url('http://stitch.embl.de/images/infobox.lowerright_pc.png') no-repeat right bottom;">
<div style="background:url('http://stitch.embl.de/images/infobox.upperleft_pc.png') no-repeat left top;">
<div style="background:url('http://stitch.embl.de/images/infobox.lowerleft_pc.png') no-repeat left bottom; padding: 3px 12px 12px 12px;">
<form action='show_input_page.pl' method='post'>
<div style='text-align:left;'>
<input type='hidden' name='UserId' value='xrrQRRQ9bQjQ'/>
<input type='hidden' name='sessionId' value='k8xp6Dr4j42R'/>
You have not provided any input.
<div style='width:100%;text-align:center;'><input type='submit' value="&nbsp;OK&nbsp;"/></div>
</div>
</form>
</div></div></div></div></td></tr></table></td></tr></table></div>
</td>
</tr>
</table>
</td></tr></table></div>
</body>
</html>
Was it helpful?

Solution

Your URL already contains GET parameters:

stitch_results_url = 'http://stitch.embl.de/cgi/show_network_section.pl?identifier=802634%20802659&additional_network_nodes=0&chemicalmode=10.5&input_query_species=7955&interactive=yes&internal_call=1&limit=10&minprotchem=1&network_flavor=actions&required_score=400&sessionId=0ZKGxMnojGDg&targetmode=proteins&userId=Ri1Zp7hT9L9G'

but the form action attribute does not use these parameters:

<form action="/cgi/show_network_section.pl" id='standard_parameters' method='post'>

Remove these from the URL. Next, don't URL encode everything yourself, leave that to requests:

import requests

stitch_results_url = 'http://stitch.embl.de/cgi/show_network_section.pl'
# the custom limit of the interaction shown
interaction_shown_custom_limit = 400
# parameters for active prediction methods
neighborhood = 'off'
gene_fusion = 'off'
co_occurrence = 'off'
co_expression = 'off'
experiments = 'on'
databases = 'on'
textmining = 'on'
# required confidence score
required_confidence = 'high confidence (700)'

# Input parameters we are going to send
parameters = {
    'channel1': neighborhood,
    'channel2': gene_fusion,
    'channel3': co_occurrence,
    'channel4': co_expression,
    'channel5': experiments,
    'channel6': databases,
    'channel7': textmining,
    'required_score':required_confidence,
    'custom_limit':interaction_shown_custom_limit
}

req = requests.post(stitch_results_url, params=parameters)

If the form is still not accepted this way, it may be that the JavaScript on that page adds additional information to the form, the form requires the HTTP Referrer header to be set, or the site stores additional information in cookies or a cookie-keyed session.

Use a Session object to pass along any cookies the site sets, add a Referrer to the request headers, and verify with your browser development tools you didn't miss some POST fields.

I note that you are not including the UserId and sessionId parameters, for example. You'll need to include all parameters named in the query portion of the URL for your POST request to work.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top