HTTP Post on Google App Engine only works half of the time

Question

I have used URLFetch class to do HTTP Post in Google App engine to my php file (on some different website).

This is what I have

try{
byte[] data = ("EMAIL=bo0@gmail.com&TITLE=evolution&COMMENT=comments&PRICE=5000").getBytes("UTF-8");
URL url = new URL("http://www.box.com/nost.php");
HTTPRequest request = new HTTPRequest(url, HTTPMethod.POST);
request.setPayload(data);
HTTPResponse response = URLFetchServiceFactory.getURLFetchService().fetch(request);
}

now this seems to work perfectly fine when I deploy locally. However, when I publish on google app engine, this only works half of the time. (ie. even with exact same data, i press post, more than once, it can be either error or success)

Since I am not changing any data, this seems to be totally random. Can anyone give me advice why this is happening?

EDIT: this is a working code It seems like my php was blocking http request that gae is making , some of the time. I fixed by using recursion inside catch block to keep trying until succeed.

try {
    URL url = new URL("http://www.mywebsite.com/myfile.php");
    HttpURLConnection connection = (HttpURLConnection) url.openConnection();
    connection.setDoOutput(true);
    connection.setRequestMethod("POST");

    OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream());
writer.write("EMAIL="+URLEncoder.encode("sam@gmail.com", "UTF-8")+
"&TITLE="+URLEncoder.encode("myTitle", "UTF-8")+
"&PRICE="+URLEncoder.encode(String.valueOf(10000), "UTF-8"));
writer.close();

if (connection.getResponseCode() == HttpURLConnection.HTTP_OK) {
// OK
return true;
} else {
// Server returned HTTP error code.
//keep retrying until you succeed!! fighting!!
return newPost(postComponent);
}
} catch (MalformedURLException e) {
e.printStackTrace();
return false;
} catch (IOException e) {
e.printStackTrace();
return false;
}
catch (Exception e){
e.printStackTrace();
//ah dam probably server is down so you went stack overflow I give up
return false;
}

Answer 1

Instead of using the URLFetch service, I'd strongly recommend that you use java.net instead.

In my experience, this is the way to go. I've setup test cases before where I loop through a collection and make large numbers of POST requests to a remote server and then write the count and output to the browser so I can see how many succeeded and how many failed.

The results were interesting. I saw exceptions in my logs, and I thought to myself, "Oh great, more wasted time trying to figure out why the network sucks!". Well, I looked at my results for a hundred and a thousand test cases, and all were a success.

After digging into the exceptions, it was clear that the code actually tries to make the request a 2nd time if it, for instance, gets a 500 response from the server! This is why all my cases were successful, despite some 500 errors!

This was exactly what I was looking for, since it avoided the need for me to write my own error handlers for server issues or minor network issues, the library took care of that for me.

java.net - HTTPURLConnection has proven to be quite reliable when sending data to a server:

The following example makes an HTTP POST request to a URL with some form data:

import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLEncoder;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.IOException;
import java.io.OutputStreamWriter;

// ...
        String message = URLEncoder.encode("my message", "UTF-8");

        try {

            // this is your url that you're posting to
            URL url = new URL("http://www.box.com/nost.php");

            // create and open the connection using POST 
            HttpURLConnection connection = (HttpURLConnection) url.openConnection();
            connection.setDoOutput(true);
            connection.setRequestMethod("POST");

            OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream());

            // this is your data that you're sending to the server! 
            writer.write("EMAIL=bo0@gmail.com&TITLE=evolution&COMMENT=comments&PRICE=5000");

            writer.close();

            // this is where you can check for success/fail. 
             // Even if this doesn't properly try again, you could make another 
              // request in the ELSE for extra error handling! :)        
            if (connection.getResponseCode() == HttpURLConnection.HTTP_OK) {
                // OK
            } else {
                // Server returned HTTP error code.
            }
        } catch (MalformedURLException e) {
            // ...
        } catch (IOException e) {
            // ...
        }

Remember, you're not making a request to a php file, you're making a request to a server. The file type makes no difference as it's all HTTP under the hood. Hope this helps!

I edited the example from Google so that it uses your remote endpoint and your query string data.