Java HttpURLConnection blocked at work, but not Firefox

Question

Good afternoon,

I have an issue concerning an HttpURLConnection and the internet restrictions at work...

What I am trying to do:

I am trying to write a program that connects to the site http://www.epexspot.com and reads the peak and base product price histories for electricity.

---

Why I am trying to do this:

Up to now, the collection of the prices has been done manually, which is a tedious procedure. Thus, I wanted to automate this with a little program.

---

What I have done so far:

I wrote a Java (JDK7u21) program making use of an HttpURLConnection, trying to contact the homepage and fetching the sent response; here you can see pretty much what I wrote:

HttpConnector.java

package network;

import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.ArrayList;

public class HttpConnector {
String urlParameters, method;
URL url;
HttpURLConnection conn;
BufferedReader in;

public HttpConnector(String host, String method) throws IOException{
    if(!host.startsWith("http://") && !host.startsWith("https://"))
        host = "http://" + host;

    this.method = method;
    urlParameters = "";
    url = new URL(host);
}

public HttpConnector(String host, String method, String parameters) throws IOException{
    if(!host.startsWith("http://") && !host.startsWith("https://"))
        host = "http://" + host;

    this.method = method;
    urlParameters = parameters;
    url = new URL(host);
}

public void openConnection() throws IOException{
    conn = (HttpURLConnection) url.openConnection();
    conn.setRequestMethod(method);
    conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20100101 Firefox/21.0");
    conn.setRequestProperty("Host", url.getHost());
    conn.setRequestProperty("Connection", "keep-alive");
    if(urlParameters!="" && urlParameters!=null) 
        conn.setRequestProperty("Content-Length", Integer.toString(urlParameters.getBytes().length));
    conn.setRequestProperty("Accept-Language", "de-de,de;q=0.8,en-us;q=0.5,en;q=0.3");
    conn.setRequestProperty("Accept-Encoding", "deflate");/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
    conn.setUseCaches(false);
    conn.setDoInput(true);
    conn.setDoOutput(true);
}

public void sendRequest() throws IOException{
    if(method == "POST"){
        DataOutputStream out = new DataOutputStream(conn.getOutputStream());
        out.writeBytes(urlParameters);
        out.flush();
        out.close();
    }
}

public ArrayList<String> read() throws IOException{
    if(conn.getResponseCode()>226 || conn.getResponseCode()<200){
        try{
            in = new BufferedReader(new InputStreamReader(conn.getErrorStream()));
        }catch(NullPointerException e){
            in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
        }
    }else{
        in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
    }
    ArrayList<String> resp = new ArrayList<String>();
    String respTmp;

    while((respTmp=in.readLine())!=null){
        resp.add(respTmp);
    }
    return resp;
}

public void close(){
    if(conn!=null) conn.disconnect();
}

public ArrayList<String> communicate() throws IOException{
    ArrayList<String> resp = new ArrayList<String>();
    try{
        openConnection();
        sendRequest();
        resp=read();
    }catch(Exception e){
        e.printStackTrace(System.err);
    }finally{
        close();
    }
    return resp;
}

}

Main.java

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.util.ArrayList;

import network.HttpConnector;


public class Main {
    public static void main(String[] args) {
        try{
            File f = new File("response.html");
            if(!f.exists()) f.createNewFile();

//          String host = "http://www.epexspot.com/en/market-data/auction/auction-table/2013-05-28/DE";
// this is where I actually need to go; google.at is merely for testing purposes
            String host = "www.google.at";
            String method = "GET";

            ArrayList<String> response = new ArrayList<String>();
            HttpConnector conn = new HttpConnector(host,method);
            response = conn.communicate();

            FileWriter fw = new FileWriter(f);
            BufferedWriter out = new BufferedWriter(fw);

            for(String resp : response){
                System.out.println(resp);
                out.write(resp+"\n");
            }

            out.flush();
            out.close();
            fw.close();
        }catch(Exception e){
            e.printStackTrace();
        }
    }

}

Short explanation: The HttpConnector connects to a given Host with given method (primarily POST or GET) and given URL parameters (which i don't use, though). It sets some Request Properties (such as the User-Agent, ...) and then tries to read the response (via the InputStream; if the response status says it has been unsuccessful, via the ErrorStream).

Main is calling the HttpConnector with a specific URL (e.g. www.epexspot.com/en/) and a specific method (either POST or GET). It then reads the response of the connection and prints it to the console aswell as a file (response.html).

---

Where my problem is:

Here at work, the traffic is kind of regulated, meaning that some homepages are blocked (just the way that they are blocked at school). So, of course if I feed some URL for a social media platform to my little program, it will spit out something like "Error 403 - contents of the page have been blocked. if you need this page for work, please contact your admin".

This, for example happens to me when I try to reach the desired page, epexspot.com - BUT: the page is NOT blocked, when I am calling it with the ordinary Mozilla Firefox (v21). On some pages, my program will work just fine, but not on most (e.g. www.google.at, www.ivb.at work just fine... whereas most other pages don't)

I have already tried to make my program act like it was Firefox concerning the Request Properties, but it has lead to no result up to now... Am I missing some Request Property or setting that could make the internet regulation software block my program, but not Mozilla Firefox?

---

So, my primary question is:

What could be the reason that my program keeps being blocked, while Firefox won't experience a blocking level anywhere near it?

I will try to contact the network admins at work, and hope that they have a solution for my program not being blocked all the time anymore, but I still wonder what could make such a significant difference between Firefox and my program.

---

Thanks in advance

Answer 1

Okay, the answer is as simple as it could be...

Firefox has been configured to use the auto-configured proxy server, so here is what I did:

I had my websites open in Firefox, did the netstat -an | find "EST" trick, figured out what the proxy's address (and port) was and made my program use them with the lines

System.setProperty("http.proxyHost", proxyAddress); and System.setProperty("http.proxyPort", "8080");

That fixed the problem for me...

Thanks, jtahlborn for this hint!

Edit:

Making use of the ProxySelector also works pretty well; should you need it, follow this link: http://docs.oracle.com/javase/6/docs/technotes/guides/net/proxies.html