For more on how wait codes are structured, see the notes on $?. As you surmised, the exit code of Test::More is multiplied by 256 and normally that value will be the number of failed tests.
The reason the wait code is useful is that it lets you see when your code under harness exits because of a signal:
use Test::More tests => 2;
ok 1,"first";
kill 15,$$;
ok 1,"second";
Will yield a wait status of 15. Whether your scripts ever exit from a signal depends on what you are doing; I had a system that had to call a remote service that was unreliable and would see SIGPIPE errors when the other end died.