Groovy HTTPBuilder : Getting the entity content from a GZIPed Chunked response

Question

I need to send a POST request to a web server and be able to read the response sent by said server. I tried using the HTTPBuilder lib with this code :

def http = new HTTPBuilder('http://myServer/')
http.setProxy("Proxy_IP", 8080, "http")
postBody = [cmd:'e',format:'sep',c:'a',b:'b',t:'u',r:'r',kl:'lop']

    http.post( body: postBody,
               requestContentType: URLENC ){ resp ->
     HttpEntity he = resp.getEntity()
     println "${resp.getAllHeaders()}"
     println he.getContentType()
     println "${resp.getEntity().getContent()}"
    }

I get an exception when this code gets executed :

ERROR errors.GrailsExceptionResolver  - EOFException occurred when processing request: [GET] /PROJECT/home/index
Unexpected end of ZLIB input stream. Stacktrace follows:
Message: Unexpected end of ZLIB input stream
    Line | Method
->>  240 | fill      in java.util.zip.InflaterInputStream
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
|    158 | read      in     ''
|    116 | read . .  in java.util.zip.GZIPInputStream
|    138 | read      in org.apache.http.conn.EofSensorInputStream
|    184 | read . .  in java.io.InputStreamReader
|    140 | read      in java.io.Reader

I turned the debug on for wire and headers and this is what I get :

2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.wire  - >> "Accept: */*[EOL]"
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.wire  - >> "Content-Length: 160[EOL]"
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.wire  - >> "Content-Type: application/x-www-form-urlencoded[EOL]"
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.wire  - >> "Host: myServer[EOL]"
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.wire  - >> "Proxy-Connection: Keep-Alive[EOL]"
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.wire  - >> "Cookie: ASPSESSIONIDSSTTACTA=NFJKHEDRFGTKCGFEHGFKOANP[EOL]"
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.wire  - >> "Cookie2: $Version=1[EOL]"
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.wire  - >> "Accept-Encoding: gzip,deflate[EOL]"
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.wire  - >> "[EOL]"
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.headers  - >> POST myUrl HTTP/1.1
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.headers  - >> Accept: */*
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.headers  - >> Content-Length: 160
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.headers  - >> Content-Type: application/x-www-form-urlencoded
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.headers  - >> Host: myServer
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.headers  - >> Proxy-Connection: Keep-Alive
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.headers  - >> Cookie: ASPSESSIONIDSSTTACTA=NFJKHEDRFGTKCGFEHGFKOANP
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.headers  - >> Cookie2: $Version=1
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.headers  - >> Accept-Encoding: gzip,deflate
2012-01-06 16:50:12,219 ["http-bio-8080"-exec-2] DEBUG http.wire  - >> "postbody"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "HTTP/1.1 200 OK[EOL]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "Date: Fri, 06 Jan 2012 15:50:14 GMT[EOL]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "Server: Microsoft-IIS/6.0[EOL]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "X-Powered-By: ASP.NET[EOL]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "Content-Type: text/html[EOL]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "Cache-control: private[EOL]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "Via: 1.1 BAYEC-BC-20[EOL]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "Transfer-Encoding: chunked[EOL]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "Proxy-Connection: Keep-Alive[EOL]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "Connection: Keep-Alive[EOL]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "Content-Encoding: gzip[EOL]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "[EOL]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.headers  - << HTTP/1.1 200 OK
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.headers  - << Date: Fri, 06 Jan 2012 15:50:14 GMT
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.headers  - << Server: Microsoft-IIS/6.0
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.headers  - << X-Powered-By: ASP.NET
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.headers  - << Content-Type: text/html
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.headers  - << Cache-control: private
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.headers  - << Via: 1.1 BAYEC-BC-20
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.headers  - << Transfer-Encoding: chunked
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.headers  - << Proxy-Connection: Keep-Alive
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.headers  - << Connection: Keep-Alive
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.headers  - << Content-Encoding: gzip
[Date: Fri, 06 Jan 2012 15:50:14 GMT, Server: Microsoft-IIS/6.0, X-Powered-By: ASP.NET, Content-Type: text/html, Cache-control: private, Via: 1.1 BAYEC-BC-20, Transfer-Encoding: chunked, Proxy-Connection: Keep-Alive, Connection: Keep-Alive, Content-Encoding: gzip]
Content-Type: text/html
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "226[EOL]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "[0x1f]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "[0x8e]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "[0x2]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "[0x0]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "[0x0][0x0][0x0][0x0][0x4][0xb]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "[0x9c]S[0xc1][0x8e][0xda]0[0x10]G[0xbd]W[0xda]ps[0xe9][0xa5][0xc1]E[0xd5][0xaa]K[0xeb][0xa4]Zh[0x16]![0x1]BI[0xee][0xc8]$[0x3][0xb1]j[0x92]`O[0x10]|[0xf2][0xee]W[0xd4][0xb1][0x3][0x82][0xec][0xa5][0xad][0x1e4]9[0xf6]x[0xe6]y[0xde][0x9b][0x99][0x87][0xf][0xc84][0x10][0xf6][0xd1][0xf7][0xf9]N[0xf1]2[0x17][0x8fa]H [0x8][0xfb]Z[0x93][0x1c][0xcc]'[0xb9][0xf0][0xfd][0xf0][0xa1][0xf5]"v[0xb5]K[0x1b][0x93][0xce][0xd2]y[0x14][0xc6][\r][\n]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "Y+[0xd8]*80[0xea]l[0xd6][0x85]IQ[0xfe]&[\n]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "d[0xe0]i<K[0xd0][0x5][0x034]z[0xa4]0[0xa23e][0x81]G[0xc7]i[0x9a][0xac][0x9e][0xa7]QB[0xed][0xe5] [0xd3][0xda]#x[0xae]![0xf0][0x10]NH[0xdb][0xb3]y[0x95][0x15][0xb8][0x97][0xe6][0xdf][0xbe]W[0x0][0xcf][0xbb][0xed][0x1e][0x90][0x93][0x2][0xb1][0xf6][0xe1][0xd0][0x88]c[0xe0]M[0xcaa][0x12][0xa1]D?5[0x8][0x1e][0xc9][0xdc][0xa9][0x83]j!~[0x90][0xac][0xe0]J[0x3][0x6]BW[0xfe][0xd3][0xd3][0xe3][0xc8][0x1f]z[0x1d][0x18][\n]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "[0x94][0x10][0xc6]fz[0xaf][0xba][0x91][0xc68][0xd1]P&[0x92][0xf9b][0xbc][0xf][0xcdd][0x1b][0x2][0xa3][0xee][0xd6]1[0xa2]7)l[0xaa][0xfc][0xdc][0x1][0x14]_[0xef][0xa3]?[0xc1][0xe9]53[0xb2]T[0xe5]=[0x18][0x19]~[0xf9]6[0x1a]=2j[0x2][0x1c] [0xcf2][0x8d]Q{S[0xa9][0x1c]T[0xe0][\r]M[0xe6][0x92]k[0x1d]xY[0xd5]Hch[0x94][0xcbd][0xe6]M[0x97](a[0xa8]B[0x86]9[0xc9][0xee][0x9d][0x0][0x5]([0xf0]B[0xa6]QU[0xe5].L[0x5]*`[0xb4];[0x19][0x6]y[0x1b][0xe5]r[0xb5][0x9c5]fF;[[0x85]V[0xc1][0xef][0x94]n[0x10][0xf5]`[0x83][0xc1a][0xd4]Qd[0xa0][0x7][e[0xcb][0x93]L[0xe2][0xd9]*M(o[0xb0][0x82][0x13]d[0xfa] [0x7]\[0xd7]?[0xcb]foj[0x1d]82^[0xb8][0xfc]5#[0x1a][0xb8][0xca][\n]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "Fyx[0xed][0x95][0xeb]#wX/[0xf1]J[0xbf][0xc1]Y[0x99][0xec]dsq[0xe2]xPT[0x17]"f[0xf7][0x1d7]b<7h[0x84]|[0xaf][0xc6]4ZF[0xf1]l[0xf2][0x9f][0xa8][0xb1]i;[0xdb][0x1d]}[0x91]5H[0xc8]L[0xf][0x9][0x8d][0xa2]4[0x9b][0xac][0xca][0xa1][0xed]Fe[0x1a]k[0xab][0xaa]=9[0xae][0xbb][0xe3]NUM[\r][0x8d][0x19][0x1e][0xfd].[0x9]j[0xb]!Z\(;[0x1d][0xba]'o[0x90][0xaf][0xec]>;[0x8f]q[0xec]B[0xc7][0xe9]z61[0x93][0xd3]3[0x8f]&[0xd1]4J[0xa3]y[0xcf][0x9c][0xbc][0xc4][0xeb]h[0x9c9][0x92][0x9e][0xd9][0x80],V[0xf3]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "[0xa4]on[0xbcd][0x17][0xc9][0xaf2]o[0xc1e][0x1b][0xdc][0x16][0xa5][0x3][0xca1]7SB[0xdb]9t[0x489][0xfd][0x1][0x0][0x0][0xff][0xff]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "[\r]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "[\n]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "0[EOL]"
2012-01-06 16:50:15,687 ["http-bio-8080"-exec-2] DEBUG http.wire  - << "[EOL]"

So I guess the problem comes from the fact that the response is sent with :

Transfer-Encoding: chunked
Content-Encoding: gzip

Is there another way to do this ? Am I doing it wrong ? I tried googling for some time and did not find a good lead. It seems to me that the HTTPBuilder lib is getting the entity content after the end of the closure but once I am out of that closure, I cannot access the entity content (exception is : Cannot access closed stream)

I really would appreciate help on this one ;)

Answer 1

Try using HTTPBuilder's setContentEncoding() method.

http.contentEncoding = ContentEncoding.Type.GZIP

That should solve your problem.

Answer 2

hm. not sure, but do you need to send the response chunked?

AFAIK, chunked means that the server sends the response in several pieces. The exceptions looks like a part of the technology stack tries to unzip the content before you recived all pieces.

So one workaround could be to send the response not chunked