Because it's binary you don't have newlines in the file. You can use slurp mode to read the whole file in one go. Looking at the spec for pgm the header values are separated by whitespace which could include spaces and newlines and tabs. I had a crack at parsing it using regex and it seems to work ok.
This example should get you started..
#!/usr/bin/perl
open(TEST, "baboon.pgm") or die "can't open \n";
binmode(TEST);
$/ = undef; # Slurp mode!
my $data = <TEST>;
close(TEST);
$data =~ /(..)\s([\d]+)\s([\d]+)\s([\d]+)/;
print "magic is $1\n";
print "width is $2\n";
print "heigh is $3\n";
print "maxval is $4\n";