Mojolicious REST + IOLoop concurrent service

Question

First of all I'm a newbie in Perl and Mojo so I'll do my best with this.. :D

The problem is that I need to work in parallel, so this service need to process several calls every second. This service will allow other services to connect over REST in order to download and process files. I'm doing some tests over this idea without success.

I've been testing around Mojo::IOLoop but I think I've misunderstood some concepts about that because every implementation using loops and timers goes absolutely wrong...

Actually my workflow goes over here:

(1) Get REST call

(2) Verify url provided over mongodb

(3) Download file associed to that mongo object.

So actually this is working but I can't see how to implement this in order to work in parallel...

#!/usr/bin/env_perl
use Mojolicious::Lite;
use MongoDB;

my $mongo     = MongoDB::MongoClient->new(host => 'xxxx.mongohq.com',port => 'xxxxx',     username => 'xxxxx', password => 'xxxxx', db_name => 'xxxxx');
my $database   = $mongo->get_database( 'xxxxx' );
my $collection = $database->get_collection( 'xxxxx' );

any [qw/ get post /] => '/:action/:objid' => sub {

  my $self = shift;

  #Parsing objectid...
  my $objectid  = $self->param('objid');
  my $id = MongoDB::OID->new(value => $objectid);

  my $all = $collection->find({_id => $id});
  my $dts = $all->next;

  download($dts->{parsedoc}->{url},"final_name.xml");

};

app->start;

sub download {
  say "Starting new download..";  
  my $ua = Mojo::UserAgent->new(max_redirects => 5);
  my $tx = $ua->get($_[0]);
  $tx->res->content->asset->move_to($_[1]);
  say "Download ends."; 
}

Which is the best way to implement IOLoop over here? I've been using some mojo timer examples but maybe I need to reorder my app flow?? Thanks in advance.

Answer 1

As I mention in my comment, Mojo::IOLoop doesn't concern with parallelism but with event loops. This means that operations which may take a long time may be started and then continued only once complete without blocking the other processes. In your example both the call to Mongo and the download are these types of long-running processes which one may want to put in the event loop. Mojolicious::UserAgent is non-blocking when used with a callback (the action called once the operation is complete). To do the same with Mongo you will need to be sure your Mongo library can be used in a non-blocking fashion.

I have mocked up a little example which ignores the Mongo part (exercise for the reader) and uses the UserAgent in a non-blocking fashion.

#!/usr/bin/env perl
use Mojolicious::Lite;

helper download => sub {
  my ($c, $url, $target) = @_;
  my $ua = $c->app->ua->max_redirects(5);

  say "Starting new download ($url) ..."; 
  $ua->get( $url, sub {
    my ($ua, $tx) = @_;
    say "Finished download ($url) ...";
    $tx->res->content->asset->move_to($target);

    say "File saved ($url --> $target)";
    $c->render( text => "File saved ($url --> $target)" );
  });
};

any [qw/ get post /] => '/:action/:objid' => sub {
  my $self = shift;
  $self->render_later; # prevent auto-render

  #Parsing objectid
  my $objectid  = $self->param('objid');
  #verify url here ...

  # example url
  my $url = 'http://www.google.com';

  $self->download($url,"final_name.xml");
};

app->start;