How to Authenticate Against Website Using Apache HttpClient 4.2.1 or >?

Question

I'm new to http programming and I'm attempting to authenticate against a website that doesn't use an API and I'm having some trouble. I found a few other questions that seemed to be similar to mine, but none had an answer that worked for me.

I've tried several different approaches but haven't found one that works yet. Actually, I had it work for me once, but didn't check in that code (I know - what was I thinking?). I haven't been able to get back to that working version again. Here are a few things that I've tried so far:

    HttpClient client = new DefaultHttpClient(); //or any method to get a client instance, with a 'threadsafe' connection manager or otherwise
    Credentials credentials = new UsernamePasswordCredentials(userName, passwd);
    ((DefaultHttpClient) client).getCredentialsProvider()
                                .setCredentials(AuthScope.ANY, credentials);

// website is defined as my target website & this constitutes the valid login URL
    HttpPost post = new HttpPost(https + website + "/login");
    HttpEntity entity = new StringEntity("Username="+ userName +"&Password="+ passwd);
    post.setEntity(entity);

    HttpResponse response = client.execute(post);
    EntityUtils.consume(entity);
    entity = response.getEntity();
    EntityUtils.consume(entity);

// the protected part of the site is over http after authentication succeeds
    HttpGet get = new HttpGet(http + website +"/protected");
    response = client.execute(get);
    entity = response.getEntity();
    String content = EntityUtils.toString(entity);

At this point the 'entity' I get back from the website is "error":"Unauthorized Access."

You'll notice that I have a 'Credentials' instance being passed to the HttpClient and I'm also putting the user name & password in the HttpPost entity. I tried each of these approaches separately and they both returned the "error":"Unauthorized Access." result.

I've tried the DefaultHttpClient, which uses a single thread connection manager, as well as 'ThreadSafeClientConnManager' and neither worked.

Answer 1

first try to login using this url (copy the url to textpad or something first - then edit it):

https://www.centraldispatch.com/login?uri=%2Fprotected%2F&Username=XXX&Password=YYY

Just replace the XXX and YYY with your real user/pass - make sure it works.

Then use:

List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(1);
nameValuePairs.add(new BasicNameValuePair("Username","Your username"));
nameValuePairs.add(new BasicNameValuePair("Password","Your password"));
post.setEntity(new UrlEncodedFormEntity(nameValuePairs));
HttpResponse response = client.execute(post);
BufferedReader rd = new BufferedReader(new InputStreamReader(
   response.getEntity().getContent()));
   String line = "";
   while ((line = rd.readLine()) != null) {
      System.out.println(line);
   }

Answer 2

Did you set the User-Agent filed?Like that:

get.setHeader("Content-Type", "text/html");
get.setHeader("User-Agent","Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.0; .NET CLR 1.1.4322; .NET CLR 2.0.50215;)");
get.setHeader("Accept-Charset", Chareset+";q=0.7,*;q=0.7");//"utf-8;q=0.7,*;q=0.7");

And maybe you should get the login page firstly,to use the cookie.

If all this don't work,you shloud use some tools like firebug+firefox to track the network process,and emu them step by step.That should work.

Websites check some fields,I think this is definitely the reason.