How to post files to a multipart/form-data?

Question

I want to upload and submit files to the form on http://www.broadinstitute.org/cmap/newQuery?servletAction=querySetup&queryType=quick (need log-in). The form looks as follows

<form name="form1" enctype="multipart/form-data" method="post" action="newQuery">
    <input type="hidden" name="servletAction" value="quickQuery">
        <div class="formTable">
            <div class="row">
                <span class="label" style="width: 100px;">up tag file:</span>
                <span class="field"><input type="file" name="ups" size="30" accept="grp"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<font size="-1"><a href="#" onClick="window.open('help_topics_frames.jsp?topic=tag list', 'helpTopicsWindow', 'scrollbars,resizable,height=600,width=700')">tag file help</a></font></span>
            </div>
            <div class="row">
                <span class="label" style="width: 100px;">down tag file:</span>
                <span class="field"><input type="file" name="dns" size="30" accept="grp"></span>
            </div>
            <div class="row">
                <span class="label" style="width: 100px;">&nbsp;</span>
                <span class="navigation"><input type="button" onClick="submitForm()" name="submitButton" value="execute query"></span>
            </div>
        </div>
</form>

First, from the answer https://stackoverflow.com/a/22547541/651779 I get the cookie with the log-in credential

import http.cookiejar
import urllib
from bs4 import BeautifulSoup
import requests

submit_signature_url = 'http://www.broadinstitute.org/cmap/newQuery?servletAction=querySetup&queryType=quick'
login_url = 'http://www.broadinstitute.org/cmap/j_security_check'

login_values = urllib.parse.urlencode({'j_username': 'example',
                                       'j_password': 'example', 
                                       'submit': 'sign in'})

payload_submit_signature = bytes(login_values, 'ascii')

cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(
    urllib.request.HTTPRedirectHandler(),
    urllib.request.HTTPHandler(debuglevel=0),
    urllib.request.HTTPSHandler(debuglevel=0),
    urllib.request.HTTPCookieProcessor(cj))

resp_1 = opener.open(submit_signature_url) #First call to capture the JSESSIONID
resp = opener.open(login_url, payload_submit_signature)

This works correctly. Now I want to post the files to the form. I tried using this answer https://stackoverflow.com/a/12385661/651779

# changed after Brett Lempereur's answer
values = {'ups':open(r'path\to\up.grp','rb'),
          'dns':open(r'path\to\down.grp','rb')}

submit_signature_url = 'http://www.broadinstitute.org/cmap/newQuery?servletAction=querySetup&queryType=quick'
req = requests.post(submit_signature_url, files=values, cookies=cj)
soup = BeautifulSoup(req.text)
print(soup.prettify())

This prints

Requested URL:  http://www.broadinstitute.org/cmap/newQuery
java.lang.NullPointerException

If you are logged in and go to http://www.broadinstitute.org/cmap/newQuery in your browser you get the same. If I instead use data instead of files in the requests.post: req = requests.post(submit_signature_url, data=values, cookies=cj), it prints the html of the page containing the form, so it did not post the form.

How can I post to multipart/form-data?

---

For an example up-file, copy the following into a file a file and call it up.grp

205258_at
221369_at
205751_at
212954_at
219914_at
206703_at
207352_s_at
203548_s_at
203549_s_at
210382_at
212110_at
213805_at
213935_at
218739_at
219737_s_at
219738_s_at
204131_s_at
204132_s_at
210655_s_at
217399_s_at
206791_s_at
206792_x_at
211818_s_at
221523_s_at
221524_s_at

And for an example down file, copy the following in a file and call it down.grp

204725_s_at
211063_s_at
219921_s_at
200618_at
219701_at
220342_x_at
220926_s_at
201323_at
204692_at
221956_at
222017_x_at
90610_at
217755_at
207843_x_at
209366_x_at
215726_s_at
201827_at
201185_at
212411_at
201692_at
214484_s_at
202910_s_at
202381_at
200663_at
201459_at
219770_at
220853_at
201349_at
207015_s_at
207016_s_at
212338_at
220330_s_at

Answer 1

To hazard a guess, the current version of the requests module expects its parameter to have one of the following formats:

files = {"name": file-handle, ...}
files = {"name": (file-name, file-handle, content-type,), ...}

Have you tried replacing your creation of the files dictionary with something along the lines of:

values = {'ups': open(r'path\to\up.grp', 'rb'), 'dns': open(r'path\to\down.grp', 'rb')}

Your code is technically but not semantically correct in that some file data will be sent to the site, but it will actually be the contents of the string literal r'path\to\up.grp'.

From reading the form, you should also pass the dictionary {"servletAction": "quickQuery"} as the data parameter to the requests.post call. In addition, the URL that you post to should be http://www.broadinstitute.org/cmap/newQuery, without the current query string that you've used in the original code.