Digest::SHA and utf8

Question

I have utf8 data which I pass to DIGEST::SHA on the following manner :

my $sha1=Digest::SHA->new;  

foreach (@fields ) {  $sha->add($record)       }
        print $sha1->hexdigest;

but i get unexpected behavior and every time i get different digest for the same record i read on http://metacpan.org/pod/Digest::SHA that there are side effects of using Digest SHA with UTF8 i trying the following metigation but both not works :

{use bytes;

foreach (@fields ) {  $sha->add($record)       }
        print    $sha1->hexdigest;
 }

and

 use Encode qw(encode_utf8);
  foreach (@fields ) {  $sha->add((encode_utf8($record))       }
        print    $sha1->hexdigest;

any help will be appreciated thanks.

Answer 1

Without seeing your actual data (e.g. use Data::Dumper; $Data::Dumper::Useqq=1; print Dumper $record;) and what SHA you expect, it is hard to help.

That said, you may find the easiest way to get a consistent SHA is to force the json encoding to output only ASCII and make sure it always puts hash elements in a consistent order:

use JSON;
$serialized = JSON::to_json( $data_structure, { 'ascii' => 1, 'canonical' => 1 } );