Apache HTTP Compoonents size limit response?

StackOverflow https://stackoverflow.com/questions/14590319

Question

I connect to server that juts returns non-finite stream of "A". I.e: this

$ nc 1.2.3.4 9001
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA^C

seems apache http component library just gets entire response from server and then parses. how to limit the size so JVM wont crash?

This is my code:

w/ fluent API:

public static void main(String[] args) throws ClientProtocolException, IOException {
    Request.Get(url).execute().discardContent();
}

w/ low level API:

public static void main(String[] args) throws ClientProtocolException, IOException {
    String url = "http://1.2.3.4:9001/";
    DefaultHttpClient httpclient = new DefaultHttpClient();
    System.out.println("A");
    HttpGet request = new HttpGet(url);
    System.out.println("B");
    HttpResponse hr = httpclient.execute(request);
    System.out.println("C");
}

gets me this:

A
B
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at org.apache.http.util.ByteArrayBuffer.expand(ByteArrayBuffer.java:62)
    at org.apache.http.util.ByteArrayBuffer.append(ByteArrayBuffer.java:92)
    at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:278)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
    at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
    at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
    at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
    at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:712)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:517)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
    at com.dudebro.WhatHappen.main(WhatHappen.java:103)

Note related question is irrelevant, i don't even ask API for response body and it still crashes.

Was it helpful?

Solution

You can try setting the following parameters:

MAX_LINE_LENGTH = "http.connection.max-line-length"
MAX_HEADER_COUNT = "http.connection.max-header-count"

I think the default for the first one is -1 which means no limit.

OTHER TIPS

What happens is clear from the stack trace: BufferedReader.readLine() keeps growing its buffer since it can't find a line terminator to start parsing the response status line, until the JVM runs out of memory. You can wrap your code in a try block and catch OutOfMemoryError, or even submit a patch to Apache maintainers, but I don't think you should really take this issue into account for a real-world application.

That server is not HTTP-compliant because the infinite stream of "A"s isn't definitely a valid HTTP reponse status-line. Usually servers are designed to be liberal for what they accept and strict for what they send, ie they should take into account that there may be uncompliant clients into the wild.

On the other side, often client developers assume that the server is well-implemented because, if it isn't, then the business running the server won't get any client...

In this case it's clear that the target server doesn't provide any valuable service, since it isn't even protocol-compliant, so nobody can take anything from it, Google can't index it, so who would spread the public URL to such an application? I think it's fine that the client crashes on your development machine because it tells that the server program is not working. If it's under your control you can fix it, otherwise you'll switch service provider. Again, this is not going to happen in the real world, because such a bad service will crash all of its clients and nobody will use it.

In the comments you said it's a server application that accept user-submitted URLs. In that case, you should really set a limit for the line length to avoid that a malicious user brings your server down.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top