Yes you are probably doing something wrong, or maybe your input photos are too similar.
You should start with one of the tutorials on using FaceRecognizer, such as in the OpenCV official tutorials or Chapter 8 of the book "Mastering OpenCV". And then to improve your recognition accuracy, follow the recommendations at "http://www.answers.opencv.org/question/15362/opencv-and-face-recognition/" and "http://www.answers.opencv.org/question/5651/face-recognition-with-opencv-24x-accuracy/".
And for further questions about OpenCV, you should post them on "answers.opencv.com" instead of StackOverflow, since that site has official support!