HtmlUnit proxy issue

Question

It is about WebDriver, particularly HtmlUnitDriver and FirefoxWebDriver

I used the same proxy settings for both HtmlUnitDriver and FirefoxWebDriver, but only FirefoxWebDriver works.

What I got with HtmlUnitDriver was "Access denied" from the proxy server. I got a blank page when I didn't use proxy settings. I don't think it has anything to do with the username or password, because I got the same error if I didn't set the username or password.

The following is my code, any idea would be appreciated. Thanks!

public WebDriver createHtmlUnitWebDriver() {
    HtmlUnitDriver htmlUnitDriver = new HtmlUnitDriver() {
        @Override
        protected WebClient modifyWebClient(WebClient client) {
            client.setProxyConfig(new ProxyConfig(PROXY_HOST, PROXY_PORT));

            DefaultCredentialsProvider credentialsProvider = new DefaultCredentialsProvider();
            credentialsProvider.addCredentials("username", "password");
            credentialsProvider.addProxyCredentials("username", "password", PROXY_HOST, PROXY_PORT);
            client.setCredentialsProvider(credentialsProvider);

            return client;
        }
    };

    htmlUnitDriver.setProxy(PROXY_HOST, PROXY_PORT);
    htmlUnitDriver.setJavascriptEnabled(true);

    return htmlUnitDriver;
}

public WebDriver createFirefoxWebDriver() {
    FirefoxProfile firefoxProfile = new FirefoxProfile();

    firefoxProfile.setPreference("network.proxy.type", 1);
    firefoxProfile.setPreference("network.proxy.http", PROXY_HOST);
    firefoxProfile.setPreference("network.proxy.http_port", PROXY_PORT);
    firefoxProfile.setPreference("network.proxy.ssl", PROXY_HOST);
    firefoxProfile.setPreference("network.proxy.ssl_port", PROXY_PORT);
    firefoxProfile.setPreference("network.proxy.no_proxies_on", "");

    return new FirefoxDriver(firefoxProfile);
}

Answer 1

Using:

HtmlUnitDriver htmlUnitDriver = new HtmlUnitDriver();
htmlUnitDriver.get("http://www.google.com");

I can see that proxy details are sent to the underlying HttpClient (In HttpWebConnection.getResponse()) method.

Can you provide your proxy settings (if it is publicly available), or you can try to directly use HttpClient, to see if it is not compatible with your proxy or not?

P.S.: it is better to use HtmlUnit user-list, as others may be interested to help as well.