List::Util - reduce - length - encoding - question

Question

Why do I get a wrong result with the first reduce example?

test.txt

__BE  
bb bbbbbbbbbbbbbbb  

aaaaaa  

test.pl

#!/usr/bin/env perl
use warnings; use 5.012;
use open ':encoding(UTF-8)';
use List::Util qw(reduce);
use Encode;

my( @list, $longest, $len );
open my $fh, '<', 'test.txt' or die $!;
    while( my $line = readline( $fh ) ) {
    chomp $line;
    push @list, split( /\s+/, $line );
    }
close $fh;

$longest = reduce{ length($a) > length($b) ? $a : $b } @list;
$len = length $longest;
say $longest; # aaaaaa
say $len;     # 6

$longest = reduce{ length(Encode::encode_utf8($a)) > length(Encode::encode_utf8($b)) ? $a : $b } @list;
$len = length(Encode::encode_utf8($longest));
say $longest;  # bbbbbbbbbbbbbbb
say $len;     # 15

$longest = $list[0];
$len = length $longest;
for my $str (@list) {
    if ( length($str) > $len ) {
        $longest = $str;
        $len = length($str);
    }
}
say $longest; # bbbbbbbbbbbbbbb
say $len;     # 15

Answer 1

AFAICS, it might even be a bug in Perl...it certainly isn't obvious that it is behaving correctly. I modified the first reduce to print diagnostics as it goes:

#!/usr/bin/env perl
use warnings; use 5.012;
use open ':encoding(UTF-8)';
use List::Util qw(reduce);
use Encode;

my( @list, $longest, $len );
open my $fh, '<', 'test.txt' or die $!;
    while( my $line = readline( $fh ) ) {
    chomp $line;
    push @list, split( /\s+/, $line );
    }
close $fh;

$longest = reduce { say "<<$a>>/<<$b>> : ", length($a), " : ", length($b);
                    length($a) > length($b) ? $a : $b } @list;
$len = length $longest;
say $longest; # aaaaaa
say $len;     # 6

$longest = reduce { length(Encode::encode_utf8($a)) > length(Encode::encode_utf8($b)) ? $a : $b } @list;
$len = length(Encode::encode_utf8($longest));
say $longest;  # bbbbbbbbbbbbbbb
say $len;     # 15

$longest = $list[0];
$len = length $longest;
for my $str (@list) {
    if ( length($str) > $len ) {
        $longest = $str;
        $len = length($str);
    }
}
say $longest; # bbbbbbbbbbbbbbb
say $len;     # 15

When run on MacOS X (10.6.5) using Perl 5.13.4, the output I get is:

<<>>/<<__BE>> : 0 : 4
<<__BE>>/<<>> : 0 : 0
<<>>/<<bb>> : 0 : 2
<<bb>>/<<bbbbbbbbbbbbbbb>> : 0 : 15
<<bbbbbbbbbbbbbbb>>/<<>> : 0 : 0
<<>>/<<aaaaaa>> : 0 : 6
aaaaaa
6
bbbbbbbbbbbbbbb
15
bbbbbbbbbbbbbbb
15

To all appearances, the first argument to the first reduce is always a zero length string, even on those odd occasions when it contains some data.

If the 'use open ':encoding(UTF-8)';' line is removed, then it behaves sanely.

<<>>/<<__BE>> : 0 : 4
<<__BE>>/<<>> : 4 : 0
<<__BE>>/<<bb>> : 4 : 2
<<__BE>>/<<bbbbbbbbbbbbbbb>> : 4 : 15
<<bbbbbbbbbbbbbbb>>/<<>> : 15 : 0
<<bbbbbbbbbbbbbbb>>/<<aaaaaa>> : 15 : 6
bbbbbbbbbbbbbbb
15
bbbbbbbbbbbbbbb
15
bbbbbbbbbbbbbbb
15

That might suggest that the bug is somewhere in the interaction of file I/O, UTF-8 encoding and List::Util. On the other hand, it could be somewhere more obscure. But my impression is that you have a test case that is reproducible and could be reported as a possible bug somewhere in Perl and its core modules.

Answer 2

I've reported this as bug in List::Util after trying to modify this program.