LWP Get Large File Download Headers Missing

Question

This post is follow on work related to LWP GET large file download. That post was regarding an error from LWP when trying to pass arguments in the header incorrectly. Now I am posting the changes I made and how I am trying to debug the approach. This discussion should be very informative for those interested in POST vs GET header formation, and what the server receives while using the CGI package. It is not information easily found on the net.

Here is my client code snip:

my $bytes_received = 0;  # vars used below are set prior to this point
my $filename = $opt{t}."/$srcfile";
open (FH, ">", "$filename") or $logger->error( "Couldn't open $filename for writing: $!" );
my $ua = LWP::UserAgent->new();
my $target = $srcfile;
my $res = $ua->get(
    $url,
    ':content_cb' => \&callback,
    'api' => 'olfs',  # Note attempted use of different types of quotes had no impact
    "cmd" => 'rfile',
    "target" => $target,
    "bs" => $bs 
    );

    print $logger->info("$bytes_received bytes received");

sub callback{
    my($chunk, $res) = @_;
    $bytes_received += length($chunk);
    print FH $chunk;
}

Here is the server snip (cgi script):

my $query = new CGI;
my $rcvd_data = Dumper($query);
print $rcvd_data;

Here is the output from a GET:

$VAR1 = bless( {
                 '.parameters' => [],
                 'use_tempfile' => 1,
                 '.charset' => 'ISO-8859-1',
                 '.fieldnames' => {},
                 'param' => {},
                 '.header_printed' => 1,
                 'escape' => 1
               }, 'CGI' );

Here is a client with a POST request:

my $ua = new LWP::UserAgent();
local $HTTP::Request::Common::DYNAMIC_FILE_UPLOAD = 1;

    my $req = 
    POST 
    $url,
    'Content_Type' => 'form-data', 
    'Content' => { 
        "api" => 'olfs',
        "cmd" => 'wfile',
        "target" => $target,
        "tsize" => $file_size,
        "bs" => $bs,
        "filename" => [ $file ] };

# HTTP::Message calls set_content, which appears to set the subroutine for content
# LWP::UserAgent 
# LWP::Protocol::file::request sends content in chunks
#

    $req->content( $req->content() );
    $logger->info("Uploading: $file");
    my $resp = $ua->request($req);

Here is the output on the server, just like before but now from the POST:

         '.parameters' => [
                            'cmd',
                            'bs',
                            'api',
                            'target',
                            'filename',
                            'tsize'
                          ],
         'use_tempfile' => 1,
         '.tmpfiles' => {
                          '*Fh::fh00001random23' => {
                                                      'info' => {
                                                                  'Content-Type' => 'text/plain',
                                                                  'Content-Disposition' => 'form-data; name="filename"; filename="random23"'
                                                                },
                                                      'name' => bless( do{\(my $o = '/usr/tmp/CGItemp33113')}, 'CGITempFile' ),
                                                      'hndl' => bless( \*Fh::fh00001random23, 'Fh' )
                                                    }
                        },
         '.charset' => 'ISO-8859-1',
         '.fieldnames' => {},
         'param' => {
                      'cmd' => [
                                 'wfile'
                               ],
                      'bs' => [
                                'buffer1'
                              ],
                      'api' => [
                                 'olfs'
                               ],
                      'target' => [
                                    'random23'
                                  ],
                      'tsize' => [
                                   '1073741824'
                                 ],
                      'filename' => [
                                      $VAR1->{'.tmpfiles'}{'*Fh::fh00001random23'}{'hndl'}
                    },
         'escape' => 1,
         '.header_printed' => 1
       }, 'CGI' );

In short, you can see in the POST dump the "key" / "value" pairs, ie "target => random23". In the GET dump I do not find any keys or values from what I submitted on the client side. Can that be explained, or what do I need to do so as to extract key / value pairs in the CGI script?

Answer 1

You're passing your form variables as HTTP headers.

Like I previously mentioned, if you want to build a url, you can use URI.

$url = URI->new($url);
$url->query_form(
   api    => 'olfs',
   cmd    => 'rfile',
   target => $target,
   bs     => $bs,
);