Having trouble locating memory leak in Perl

Question

I'm experiencing a memory leak (I think) in this snippet of code:

if (ref($init{'time-layout'}) eq 'HASH') {
    my %time= % {$init{'time-layout'}};
    my @times= @ {$time{'start-valid-time'}};
    if (ref($init{parameters}) eq 'HASH') {
        %param= % {$init{parameters}};
        if (ref($param{'wind-speed'}) eq 'HASH') {
            %wind= % {$param{'wind-speed'}};
            @windvalue= @ {$wind{value}};
            %temp= % {$param{'temperature'}};
            @tempvalue = @ {$temp{value}};
            %wdir= % {$param{'direction'}};
            @wdirvalue = @ {$wdir{value}};
            %hum= % {$param{'humidity'}};
            @humvalue = @ {$hum{value}};
        undef %wind;
        undef %temp;
        undef %wdir;
        undef %hum;}
    undef %param;}

This is from a sub, where the input comes from XMLin() (from XML::Simple). Specifically the input is (if it matters)

http://graphical.weather.gov/xml/sample_products/browser_interface/ndfdXMLclient.php?    lat=$lat&lon=$lon&product=time-series&begin=2004-01-01T00:00:00&end=2013-04-20T00:00:00&temp=temp&wspd=wspd&rh=rh&wdir=wdir";

where $lat and $lon are latitude and longitude.

The code takes a couple hours, to run plugging data into a SQL server, and well before it's done the instance of Perl is taking several GBs of RAM or more. I'm sure this is simple for someone but I'm, at best, a novice in Perl.

Answer 1

A quick duckduckgo search for "xml::simple memory leak" turned up the documentation for Yahoo::Search, which claims:

XML::Simple uses XML::Parser under the hood, and at least on the systems I've tested it, XML::Parser suffers from a crippling memory leak that makes it very undesirable.

It appears possible (perhaps even likely) that the memory leak may not be in your code at all.

Answer 2

XML::Simple is very ineffective on parsing large files (for me it eats 3Gb of memory for the 100Mb file). Try to use XML::Twig for your data.

Answer 3

In addition to imran's comment,

With so many XML cpan modules, it is very hard to decide which one should be used.

Looking at answers from many and reading from web, XML::LibXML looks a preferred module, rather more specific XML::LibXML::Reader for large files. Many of them haven't recommended using XML::Simple and says XML::Simple is good for small files and have issues when it comes to large xml files. XML::Twig is better(from it's documentation), but even that is having memory leak problems also. (refer XML::Twig faq)

You wanna also refer perl-xml-faq