mechanize and Ruby multipart/form-data - content transfer encoding

Question

I am trying to dispatch a multipart/form-data POST request to a remote server using mechanize 2.7.3 to automate some interactions with a remote server. Unfortunally there is no usable <form>, so I have to issue the POST directly. Luckily, mechanize recognizes what I am up to anyway, but the server doesn't accept the response:

c:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/mechanize-2.7.3/lib/mechanize/http/agent.rb:308: in `fetch': 400 => Net::HTTPBadRequest for http://REDACTED/api/resources -- unhandled response (Mechanize::ResponseCodeError)
    from c:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/mechanize-2.7.3/lib/mechanize.rb:1281:in `post_form'
    from c:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/mechanize-2.7.3/lib/mechanize.rb:502:in `post'
    from thing.rb:38:in `upload'

Here's what mechanize logging output looks like during the request in question:

D, [2014-05-13T12:40:34.580906 #3456] DEBUG -- : query: "--cNPsKCeBSrPGUwxyjMze\r\nContent-Disposition: form-data; name=\"force_create\"\r\n\r\ntrue\r\n--cNPsKCeBSrPGUwxyjMze\r\nContent-Disposition: form-data; name=\"file\"; filename=\"test.wgt\"\r\nContent-Transfer-Encoding: binary\r\n\r\nREDACTED\r\n--cNPsKCeBSrPGUwxyjMze--\r\n"
I, [2014-05-13T12:40:34.581906 #3456]  INFO -- : Net::HTTP::Post: /api/resources
D, [2014-05-13T12:40:34.581906 #3456] DEBUG -- : request-header: accept => */*
D, [2014-05-13T12:40:34.581906 #3456] DEBUG -- : request-header: user-agent => Mechanize/2.7.3 Ruby/1.9.3p392 (http://github.com/sparklemotion/mechanize/)
D, [2014-05-13T12:40:34.581906 #3456] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2014-05-13T12:40:34.581906 #3456] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2014-05-13T12:40:34.582906 #3456] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2014-05-13T12:40:34.582906 #3456] DEBUG -- : request-header: cookie => csrftoken=G4dZv6fxRMq0Y2h5yntDsJTFBeEKVFaU; sessionid=9oqufupdt6r620eyep4l4jcl6i2cxda5
D, [2014-05-13T12:40:34.582906 #3456] DEBUG -- : request-header: host => REDACTED
D, [2014-05-13T12:40:34.582906 #3456] DEBUG -- : request-header: referer => REDACTED/login
D, [2014-05-13T12:40:34.582906 #3456] DEBUG -- : request-header: content-type => multipart/form-data; boundary=cNPsKCeBSrPGUwxyjMze
D, [2014-05-13T12:40:34.582906 #3456] DEBUG -- : request-header: content-length => 409

I, [2014-05-13T12:40:34.613906 #3456]  INFO -- : status: Net::HTTPBadRequest 1.1 400 BAD REQUEST
D, [2014-05-13T12:40:34.613906 #3456] DEBUG -- : response-header: date => Tue, 13 May 2014 10:33:44 GMT
D, [2014-05-13T12:40:34.613906 #3456] DEBUG -- : response-header: server => Apache/2.2.15 (CentOS)
D, [2014-05-13T12:40:34.613906 #3456] DEBUG -- : response-header: vary => Accept-Language,Cookie,Accept-Encoding
D, [2014-05-13T12:40:34.613906 #3456] DEBUG -- : response-header: content-language => en
D, [2014-05-13T12:40:34.613906 #3456] DEBUG -- : response-header: content-type => text/plain; charset=utf-8
D, [2014-05-13T12:40:34.613906 #3456] DEBUG -- : response-header: content-encoding => gzip
D, [2014-05-13T12:40:34.613906 #3456] DEBUG -- : response-header: content-length => 37
D, [2014-05-13T12:40:34.613906 #3456] DEBUG -- : response-header: connection => close
D, [2014-05-13T12:40:34.614906 #3456] DEBUG -- : Read 37 bytes (37 total)
D, [2014-05-13T12:40:34.614906 #3456] DEBUG -- : gzip response

This is how I do it in Ruby:

def upload(file)
    agent.post(base_uri + RESOURCES_PATH, {
        :force_create => true,
        :file => File.new(file)
    }) do … end
end

Every other interaction with the site works normally (mechanize actually even logs in first, thus the cookies). The upload also works fine in a real browser – the only discernable difference is the content transfer encoding (octet-stream instead of binary). Or maybe I am missing something here? Something vital?

Also, I can't say much about the remote server in question. I only know the site is running some sort of Python-based framework (Django, iirc).

Thanks in advance,
Manuel

Answer 1

I have managed to identify the culprit here. As my development machine is Windows-based, this seems to have been an issue with mechanize (or one of its dependencies) and Windows. By specifying the b (binary) part in the second argument of File.new, the problem went away on its own. tl;dr: here's how the working code fragment now looks like:

agent.post(base_uri + RESOURCES_PATH, {
    :force_create => true,
    :file => File.new(file, 'rb')  # <-- changed
})