Google App Engine ( Java ) : URL Fetch Response too large problems
-
10-10-2019 - |
문제
I'm trying to build some sort of webservice on google apps.
Now the problem is, I need to get data from a website (HTML Scraping).
The request looks like :
URL url = new URL(p_url);
con = (HttpURLConnection) url.openConnection();
InputStreamReader in = new InputStreamReader(con.getInputStream());
BufferedReader reader = new BufferedReader(in);
String result = "";
String line = "";
while((line = reader.readLine()) != null)
{
System.out.println(line);
}
return result;
Now App Engine gives me the follwing exception at the 3th line:
com.google.appengine.api.urlfetch.ResponseTooLargeException
This is because the maximum request limit is at 1mb and the total HTML from the page is about 1.5mb.
Now my question: I only need the first 20 lines of the html to scrape. Is there a way to only get a part of the HTML so that the ResponseTooLargeException will not be thrown?
Thanks in advance!
해결책
Solved the problem by using the low level URLFetch api.
And setting the allowtruncate option to true;
Basicly it works like this :
HTTPRequest request = new HTTPRequest(_url, HTTPMethod.POST, Builder.allowTruncate());
URLFetchService service = URLFetchServiceFactory.getURLFetchService();
HTTPResponse response = service.fetch(request);
제휴하지 않습니다 StackOverflow