The unit is byte.
The best clue for this is from tools/plotframes, a tool provided by ffmpeg to generate a plot of frame sizes. Check this code snippets, which would output frame size in Kbits (pkt_size * 8 /1000).
foreach my $frame (@{$frames}) {
my $type = $frame->{pict_type};
$frame->{count} = $frame_count++;
if (not $stats{$type}) {
$stats{$type}->{tmpfile} = File::Temp->new(SUFFIX => '.dat');
my $fn = $stats{$type}->{tmpfile}->filename;
open($stats{$type}->{fh}, ">", $fn) or die "Can't open $fn";
}
print { $stats{$type}->{fh} }
"$frame->{count} ", $frame->{pkt_size} * 8 / 1000, "\n";
}