Decode a streaming GZIP response in Scala Dispatch?

Question

Receiving a Gzipped response from an API, but Dispatch 0.9.5 doesn't appear to have any methods to decode the response. Any ideas?

Here's my current implementation, the println only prints out string representations of bytes.

   Http(
      host("stream.gnip.com")
      .secure
      .addHeader("Accept-Encoding", "gzip")
       / gnipUrl
      > as.stream.Lines(println))()

Tried to look at implementing my own handler, but not sure where to begin. Here's the relevant file for Lines: https://github.com/dispatch/reboot/blob/master/core/src/main/scala/as/stream/lines.scala

Thanks!

Answer 1

Simply abandoned Dispatch and used Java APIs directly. Disappointing, but it got the job done.

  val GNIP_URL = isDev match {
    case true => "https://url/apath/track/dev.json"
    case false => "https://url/path/track/prod.json"
  }
  val GNIP_CHARSET = "UTF-8"

  override def preStart() = {
    log.info("[tracker] Starting new Twitter PowerTrack connection to %s" format GNIP_URL)

    val connection = getConnection(GNIP_URL, GNIP_USER, GNIP_PASSWORD)
    val inputStream = connection.getInputStream()
    val reader = new BufferedReader(new InputStreamReader(new StreamingGZIPInputStream(inputStream), GNIP_CHARSET))
    var line = reader.readLine()
    while(line != null){
        println(line)
        line = reader.readLine()
    }
  }

  private def getConnection(urlString: String, user: String, password: String): HttpURLConnection = {
    val url = new URL(urlString)

    val connection = url.openConnection().asInstanceOf[HttpURLConnection]
    connection.setReadTimeout(1000 * 60 * 60)
    connection.setConnectTimeout(1000 * 10)

    connection.setRequestProperty("Authorization", createAuthHeader(user, password));
    connection.setRequestProperty("Accept-Encoding", "gzip")
    connection
  }

  private def createAuthHeader(username: String, password: String) = {
    val encoder = new BASE64Encoder()
    val authToken = username+":"+password
   "Basic "+encoder.encode(authToken.getBytes())
  }

Used GNIP's example: https://github.com/gnip/support/blob/master/Premium%20Stream%20Connection/Java/StreamingConnection.java

Answer 2

This isn't so much a solution as a workaround, but I ended up resorting to bypassing the Future-based stuff and doing:

val stream = Http(req OK as.Response(_.getResponseBodyAsStream)).apply val result = JsonParser.parse( new java.io.InputStreamReader( new java.util.zip.GZIPInputStream(stream)))

I'm using JsonParser here because in my case the data I'm receiving happens to be JSON; substitute with something else in your use case, if needed.

Answer 3

My solution just defined a response parser and also adopted json4s parser:

    object GzipJson extends (Response => JValue) {
      def apply(r: Response) = {
        if(r.getHeader("content-encoding")!=null && r.getHeader("content-encoding").equals("gzip")){
          (parse(new GZIPInputStream(r.getResponseBodyAsStream), true))
        }else
          (dispatch.as.String andThen (s => parse(StringInput(s), true)))(r)
      }
    }

So that I can use it to extract Gzip Json response as the following code:

    import GzipJson._

    Http(req OK GzipJson).apply

Answer 4

Try >> instead of >

See https://github.com/dispatch/dispatch/blob/master/core/src/main/scala/handlers.scala#L58