Getting response returns original page instead in LWP

Question

I'm trying to submit a form without filling in anything and getting the response page.

#!/usr/bin/perl -w

use LWP::UserAgent;
use HTTP::Request::Common;

$ua= LWP::UserAgent->new;
$req= POST('http://polling.newspoll.com.au/cgi-bin/polling/display_poll_data.pl?url_caller=&mode=trend&page=show_polls&question_set_id=4');

$res= $ua->request($req);
print $res->content;

But I just get the original page (with the form) printed not the response. What am I doing wrong?

EDIT: It was just a matter of finding the correct parameters to pass to the page. To figure this out I used Chrome Developer Tools on the results page, then go to network tab, then headers. SCroll down a bit and you will find all form parameters that were supplied to the page. Thanks to amon for pointing me in the right direction.

Answer 1

The POST helper does not take a full URL including query parameters. You should provide those in an arrayref, like:

my $req = POST('http://www.example.com', [ foo => 42, bar => 'baz' ]);

… as shown in the docs.

You should also note that POST parameters are not transported as query parameters inside the URL, but rather inside the body of the POST request. While some software like CGI provides an interface that abstracts away the difference, other software may not.

Please also note that all Perl scripts should use strict; use warnings;.

Another thing you should consider is proper error handling by testing is_success and using the decoded_content method, which takes care of transfer encodings.

$res->is_success
  or die $res->status_line;

print $res->decoded_content;

---

Edit

I analyzed the source, with the following results (autogenerated):

I looked at the source of the page at http://polling.newspoll.com.au/cgi-bin/polling/display_poll_data.pl, and the form has following <input>s:

• <input name="url_caller" type="hidden" value="" />
• <input name="mode" type="hidden" value="file" />
• <input name="page" type="hidden" value="Search" />
• <input name="keywords" type="text" />
• <input name="show" type="text" />
• <input src="/images/search_button.jpg" type="image" />

And the following <select> groups:

• <select name="from_date_month"></select> with values (1 .. 12)
• <select name="from_date_year"></select> with values (1998 .. 2013)
• <select name="to_date_month"></select> with values (1 .. 12)
• <select name="to_date_year"></select> with values (1998 .. 2013)
• <select name="keytype"></select> with values ("all", "any")
• <select name="state"></select> with values ("Any", "FEDERAL", "ACT", "NSW", "NT", "QLD", "SA", "TAS", "VIC", "WA")

Additionally, links with the following queries are present:

• ?url_caller=latest&state=Any&mode=file&page=Search
• ?mode=trend&page=select_category

which means that the script also responds to GET requests

I looked at the source of the page at http://polling.newspoll.com.au/cgi-bin/polling/display_poll_data.pl?url_caller=&mode=trend&page=show_polls&question_set_id=4, and the form has following <input>s:

• <input name="url_caller" type="hidden" value="trend" />
• <input name="show" size="5" type="text" />
• <input name="mode" type="hidden" value="trend" />
• <input name="question_id" type="hidden" value="2403" />
• <input name="page" type="submit" value="Display Results" />
• <input name=".cgifields" type="hidden" value="to_date_year" />
• <input name=".cgifields" type="hidden" value="to_date_month" />
• <input name=".cgifields" type="hidden" value="from_date_month" />
• <input name=".cgifields" type="hidden" value="to_date_day" />
• <input name=".cgifields" type="hidden" value="from_date_year" />
• <input name=".cgifields" type="hidden" value="from_date_day" />

And the following <select> groups:

• <select name="from_date_day"></select> with values (1 .. 31)
• <select name="from_date_month"></select> with values (0 .. 12)
• <select name="from_date_year"></select> with values ("Any Year", 1970 .. 2013)
• <select name="to_date_day"></select> with values (1 .. 31)
• <select name="to_date_month"></select> with values (0 .. 12)
• <select name="to_date_year"></select> with values ("Any Year", 1970 .. 2013)

Additionally, links with the following queries are present:

• ?url_caller=trend
• ?url_caller=trend&mode=trend&page=show_polls&question_set_id=4
• ?mode=trend&page=continue_results&question_id=2403&url_caller=trend

which means that the script also responds to GET requests

You are using the fields [url_caller => '', mode => "trend", page => "show_polls", question_set_id => 4] in your request. E.g. the field question_set_id does not exist. The value show_polls isn't present for page.

Hidden fields should not be changed, but re-submitted as they are prefilled. For the selections, pick valid arguments from the provided lists. For text input, provide a sensible value depending on the context.