IO::Lambda in Perl
-
09-10-2019 - |
Question
I've been offloaded some maintenance tasks on a couple of Perl scripts. One of the requirements is to download a couple of dozen files (HTTP) in parallel. I went looking on CPAN for the easiest solution and found this module called IO::Lambda::HTTP.
Unfortunately, I have absolutely no experience with functional programming (and beginner-level Perl experience), so while I see that all of the examples work as documented, I can't really modify any of them to suit my needs.
For example, the sample that comes with the module:
#!/usr/bin/perl
# $Id: parallel.pl,v 1.7 2008/05/06 20:41:33 dk Exp $
#
# This example fetches two pages in parallel, one with http/1.0 another with
# http/1.1 . The idea is to demonstrate three different ways of doing so, by
# using object API, and explicit and implicit loop unrolling
#
use lib qw(./lib);
use HTTP::Request;
use IO::Lambda qw(:lambda);
use IO::Lambda::HTTP qw(http_request);
use LWP::ConnCache;
my $a = HTTP::Request-> new(
GET => "http://www.perl.com/",
);
$a-> protocol('HTTP/1.1');
$a-> headers-> header( Host => $a-> uri-> host);
my @chain = (
$a,
HTTP::Request-> new(GET => "http://www.perl.com/"),
);
sub report
{
my ( $result) = @_;
if ( ref($result) and ref($result) eq 'HTTP::Response') {
print "good:", length($result-> content), "\n";
} else {
print "bad:$result\n";
}
# print $result-> content;
}
my $style;
#$style = 'object';
#$style = 'explicit';
$style = 'implicit';
# $IO::Lambda::DEBUG++; # uncomment this to see that it indeed goes parallel
if ( $style eq 'object') {
## object API, all references and bindings are explicit
sub handle {
shift;
report(@_);
}
my $master = IO::Lambda-> new;
for ( @chain) {
my $lambda = IO::Lambda::HTTP-> new( $_ );
$master-> watch_lambda( $lambda, \&handle);
}
run IO::Lambda;
} elsif ( $style eq 'explicit') {
#
# Functional API, based on context() calls. context is
# $obj and whatever arguments the current call needs, a RPN of sorts.
# The context though is not stack in this analogy, because it stays
# as is in the callback
#
# Explicit loop unrolling - we know that we have exactly 2 steps
# It's not practical in this case, but it is when a (network) protocol
# relies on precise series of reads and writes
this lambda {
context $chain[0];
http_request \&report;
context $chain[1];
http_request \&report;
};
this-> wait;
} else {
# implicit loop - we don't know how many states we need
#
# also, use 'tail'
this lambda {
context map { IO::Lambda::HTTP-> new( $_, async_dns => 1 ) } @chain;
tails { report $_ for @_ };
};
this-> wait;
}
Works as advertised, but I can't for the life of me figure out how to modify either the 'object' or 'implicit' examples to be limited to N parallel instances like the following from IO::Lambda's synopsis:
# http://search.cpan.org/~karasik/IO-Lambda/lib/IO/Lambda.pm
# crawl for all urls in parallel, but keep 10 parallel connections max
print par(10)-> wait(map { http($_) } @hosts);
Can someone show me an example of what the lambda code would look like given the above constraint (e.g limit to N instances)?
Also, what's the best way to start learning functional programming? It seems totally alien to me.
Solution
There are good other options than IO::Lambda for this task, for example AnyEvent::HTTP. See this previous SO question.
Even though I'm familiar with functional programming, the above IO::Lambda sample code looks rather hard to understand to me.