By default vim reads vimscripts (=vim commands), not input files, from stdin. That is why you cannot directly pipe man
output to vim; as others have mentioned you have to use vim -
to make vim read from stdin.
However piping vimscripts can be useful too:
vim test.txt <<EOF
:%s/[aiueo]/X/g
:wq! output.txt
EOF
The above will use vim to open test.txt, replace all vowels with X, write the results to output.txt, and quit (ignoring changes to the original file). It uses a here document but you can of course put the vim commands in a file and use vim test.txt < myscript
or cat myscript | vim test.txt
to achieve the same result.
I suspect the reason they did it this way was that you can open multiple input files but only execute one script. If input was read from stdin by default, you could only read one buffer that way.