HttpClient 4.2.2 and proxy with username/password

Question

I have a problem with HttpClient 4.2.2 and proxy with username/password. HttpClient works in this way:

1. List item
2. Send request without proxy stuff (however I set proxy parameters for each request)
3. Get response from Squid with 407 error
4. Send request with proxy stuff

It's very strange behavior, Is it possible to add info about proxy to each request? I've tried to add hardcoded "Proxy-Authorization" header to each request and it works fine, why HttpClient can't do the same?

Java code

DefaultHttpClient httpClient = new DefaultHttpClient();
httpClient.getParams().setParameter(PROTOCOL_VERSION, HTTP_1_1);
...
String proxyServer = getProxyServer();
int proxyPort = getProxyPort();
List<String> authpref = new ArrayList<String>();
authpref.add(AuthPolicy.BASIC);
httpClient.getParams().setParameter(AuthPNames.PROXY_AUTH_PREF, authpref);
String proxyUser = getProxyUser();
String proxyPassword = getProxyPassword();
CredentialsProvider credsProvider = httpClient.getCredentialsProvider();
credsProvider.setCredentials(new AuthScope(proxyServer, proxyPort), new UsernamePasswordCredentials(proxyUser, proxyPassword));
httpClient.setCredentialsProvider(credsProvider);
HttpHost proxy = new HttpHost(proxyServer, proxyPort, (proxyServer.indexOf("https") != 0) ? "http" : "https");
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
...
HttpPut put = new HttpPut(url);

/*** hardcoded header ***/
//put.addHeader("Proxy-Authorization", "Basic eHRlbmR4LmRuZXByOnF3ZXJ0eQ==");
/*** hardcoded header ***/

put.setEntity(entity);
httpClient.execute(put);
httpClient.getConnectionManager().shutdown();


[DefaultClientConnection] Sending request: PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image1.jpg HTTP/1.1
[headers] >> PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image1.jpg HTTP/1.1
[headers] >> Authorization: Basic eGRldjo0YTFmNmMwOTgyYWRkMWQ0NDg1YjRhMGE4YWMxY2JjMWNiMTA0ODc1
[headers] >> Content-Length: 2
[headers] >> Host: 172.26.27.22:8080
[headers] >> Proxy-Connection: Keep-Alive
[headers] >> User-Agent: Apache-HttpClient/4.2.2 (java 1.5)
[DefaultClientConnection] Receiving response: HTTP/1.0 407 Proxy Authentication Required
[headers] << HTTP/1.0 407 Proxy Authentication Required
[headers] << Server: squid/2.7.STABLE8
[headers] << Date: Thu, 08 Nov 2012 10:09:49 GMT
[headers] << Content-Type: text/html
[headers] << Content-Length: 1431
[headers] << X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
[headers] << Proxy-Authenticate: Basic realm="Please, enter username and password"
[headers] << X-Cache: MISS from 172.26.27.94
[headers] << X-Cache-Lookup: NONE from 172.26.27.94:3128
[headers] << Via: 1.0 172.26.27.94:3128 (squid/2.7.STABLE8)
[headers] << Connection: close
[DefaultHttpClient] Authentication required
[DefaultHttpClient] 172.26.27.94:3128 requested authentication
[ProxyAuthenticationStrategy] Authentication schemes in the order of preference: [Basic]
[DefaultHttpClient] Selected authentication options: [BASIC]
[DefaultClientConnection] Connection 0.0.0.0:63344<->172.26.27.94:3128 closed
[DefaultClientConnectionOperator] Connecting to 172.26.27.94:3128
[RequestAddCookies] CookieSpec selected: best-match
[RequestAuthCache] Re-using cached 'basic' auth scheme for http://172.26.27.22:8080
[RequestAuthCache] No credentials for preemptive authentication
[RequestProxyAuthentication] Proxy auth state: CHALLENGED
[RequestProxyAuthentication] Generating response to an authentication challenge using basic scheme
[DefaultHttpClient] Attempt 2 to execute request
[DefaultClientConnection] Sending request: PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image1.jpg HTTP/1.1
[headers] >> PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image1.jpg HTTP/1.1
[headers] >> Authorization: Basic eGRldjo0YTFmNmMwOTgyYWRkMWQ0NDg1YjRhMGE4YWMxY2JjMWNiMTA0ODc1
[headers] >> Content-Length: 2
[headers] >> Host: 172.26.27.22:8080
[headers] >> Proxy-Connection: Keep-Alive
[headers] >> User-Agent: Apache-HttpClient/4.2.2 (java 1.5)
[headers] >> Proxy-Authorization: Basic eHRlbmR4LmRuZXByOnF3ZXJ0eQ==
[DefaultClientConnection] Receiving response: HTTP/1.0 201 Created
[headers] << HTTP/1.0 201 Created
[headers] << Content-Length: 0
[headers] << Date: Thu, 08 Nov 2012 10:09:49 GMT
[headers] << X-Cache: MISS from 172.26.27.94
[headers] << X-Cache-Lookup: MISS from 172.26.27.94:3128
[headers] << Via: 1.1 172.26.27.94:3128 (squid/2.7.STABLE8)
[headers] << Connection: keep-alive
[headers] << Proxy-Connection: keep-alive

Squid log

1352370666.778      0 172.26.27.94 TCP_DENIED/407 1870 PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image1.jpg - NONE/- text/html
1352370671.429      8 172.26.27.94 TCP_MISS/201 282 PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image1.jpg proxyuser DIRECT/172.26.27.22 -
1352370671.474      0 172.26.27.94 TCP_DENIED/407 1882 PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image2.jpg - NONE/- text/html
1352370671.486      7 172.26.27.94 TCP_MISS/201 282 PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image2.jpg proxyuser DIRECT/172.26.27.22 -

With hardcoded header

There is no 407 error in the java log, and squid log

Squid log

1352370542.016      8 172.26.27.94 TCP_MISS/201 282 PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image1.jpg proxyuser DIRECT/172.26.27.22 -
1352370542.033      7 172.26.27.94 TCP_MISS/201 282 PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image2.jpg proxyuser DIRECT/172.26.27.22 -

Answer 1

The authentication process with an HTTP proxy is described in RFC2616 §14.33 and §14.34 and is exactly as you see with HttpClient. It includes:

• the proxy sending a 407 (Proxy Authentication Required) response with a Proxy-Authenticate header containing the challenge applicable to the requested resource,
• the client issuing a new request with a Proxy-Authorization header consisting of the credentials containing the authentication information.

Implementing the preemptive authentication with HttpClient is described in this tutorial (section §4.8), and requires prepopulating the authentication cache of the client object. Unfortunately, their code does not work when preauthenticating to a proxy server. It may be a bit tricky to understand how to get it right, but actually it is as simple as passing a parameter to the BasicScheme constructor:

AuthCache authCache = new BasicAuthCache();

AuthScheme basicAuthScheme = null;
if (isProxy) {
   basicAuthScheme = new BasicScheme(ChallengeState.PROXY);
} else {
   basicAuthScheme = new BasicScheme(ChallengeState.TARGET);
}

authCache.put(host, basicAuthScheme);
httpContext.setAttribute(ClientContext.AUTH_CACHE, authCache);

Answer 2

It's your AuthScope:

// You set proxyServer here vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
credsProvider.setCredentials(new AuthScope(proxyServer, proxyPort), new UsernamePasswordCredentials(proxyUser, proxyPassword));
httpClient.setCredentialsProvider(credsProvider);
// But here you are looking for the indexof https to determine if it is an SSL proxy
// is the String returned from getProxyServer() above a URL or a host name?
HttpHost proxy = new HttpHost(proxyServer, proxyPort, (proxyServer.indexOf("https") != 0) ? "http" : "https");
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);

Buried in my comment above - is the string returned by getProxyServer() a URL or a hostname?

EDIT

I think I have figured out what is going on. Basic Auth has to be sent on every request for it to work. If you are making multiple executions you are going to need to have some mechanism to cache the authentication data. By executing the client in the fashion you are right now, there is no "context" that can store that as one is created on every request.

The other item to look at is the client tutorial sections 4.7 and 4.8. If you are looking to completely eliminate the 407 error followed by a request with the BASIC authentication, then make sure you read section 4.8.

Pay very close attention to the "localcontext" variable they have defined as this acts as a state container for your client.