Login to website with redirection using WWW::Mechanize

Question

I know it sounds like a question which was already asked on forum ,however I believe that I have a slightly different scenario here..

I try to use Perl to log on to a web site which requires authentication. I already realized that the most convenient tool to accomplish this task is to use WWW::Mechanize module where it is quite easy to fill in forms and submit them.

In my case, I encounter the following difficulties:

1. it seems that there is no "form" which related to the fields I want to fill. Instead, the fields are wrapped in a table <table></table>.
2. The button itself seems to launch some javascript which I know is not supported by the Mechanize module. However, just to test the behavior of this page, I disabled the javascript in my Chrome browser and still I was able to click on the button and login to the site. So I assume that javascript is not a must here.

3. I use the following code and it seems that it doesn't matter what details(user/pass) I enter, for some reason I always follow the "Success" scenario and receive "Login successful!":

   my $mech = WWW::Mechanize->new(
    cookie_jar      => {},
    autocheck       => 1,
    onerror         => \&Carp::croak,);
   
   # Login Form
   my $response = $mech->get($url);
   if (!$response->is_success) {
       die "Login page unreachable $url: ",  $response->status_line, "\n";
   }
   
   $mech->field('Email', $Email);
   $mech->field('Password', $Password);
   $response = $mech->click();
   if ($response->is_success) {
       print "Login Successful!\n";
   } else {
       die "Login failed: ",  $response->status_line, "\n";
   }
   

As a result, the running of the script seems to be successful however when I assume that I already logged on and try to obtain any data from the site, I see (in sniffer) that I am being continuously forwarded to the "Register" page.

Any help will be very much appreciated!

Thanks in advance!

Answer 1

It seems that I have realized how to solve this problem. I used Wireshark sniffer to analyze the difference between HTTP headers which were sent when using browser and headers which were created by the mechanize object. I found out that mechanize generated "Connection" header included "TE, close" value, while browser issued "Connection" header with "keep-alive" value in it. Moreover, browser generated request included additional "Keep-Alive: 115" header. I manually added these headers to my mech object as follows:

  $mech->add_header(
  "Connection" => "keep-alive",
  "Keep-Alive" => "115");

This solved the problem and I was able to login successfully!

Thank you all for you contribution and help! It is very much appreciated!

Answer 2

If it's not in a form tag then there's a submit button somewhere which is using a GET request to send all the variables to the same (most likely) URL.

Why are you trying to emulate using a browser to click the authentication button, is there some other hidden field or something that you need access to?

In my eyes, the best way is to find out the page that is actually authenticating the user and password and use a SSL POST request to that page.

If the page has something like:

<input type="text" name="user" />
<input type="password" name="pass" />

Send over a HTTP POST request to whatever the login page is (likely the same page if no <form> is defined). The POST variables will be user=$value and pass=$other_value.

Answer 3

When you check if $response->is_success you just checking that the target site response without any HTTP errors to you. But instead you need to check if response page contain some text. For example, "Auth ERROR!" etc.

If the target site work with disabled JavaScript than this is not an issue.

Try to add x and y (coordinates of the mouse pointer) to the data submitted to the form (some site check this info).

You can use some HTTP traffic monitoring tool to check what data your browser sends to the target site (i use Firefox HTTPFox extension for that) and make a same request via WWW::Mechanize.

Also, WWW::Mechanize don't need cookie_jar. It process cookies by default.

Try this code:

$mech->get($url);
$mech->submit_form(
    form_name => 'aspnetForm',
    fields => {
        'ctl00$cph1$lg1$txtEmail' => $login,
        'ctl00$cph1$lg1$txtPassword' => $password,
    },
    button => 'ctl00$cph1$lg1$btnLogin',
);