The camera should rotate about 2 axis. The up vector and the strafe vector. Rotating about the up vector gives the side to side "look", while rotating about the strafe vector gives the up and down "look".
When you perform the rotation, you need to not only transform the direction, but also the up vector and strafe vector. You don't seem to be doing that here. At least 2 vectors must be changed after a transformation as all 3 vectors must be orthogonal to each other.
Take a look at the following as pseudocode:
public static void MouseLook(float x, float y)
{
Matrix xRot = Matrix.CreateFromAxisAngle(Vector3.Up, x * rotationSpeed);
Matrix yRot = Matrix.CreateFromAxisAngle(Vector3.Right, y * rotationSpeed);
viewMatrix = viewMatrix * yRot * xRot;
// Orthogonalize the matrix
OrthoNormalize();
}
private static void OrthoNormalize()
{
Vector3 zAxis = viewMatrix.Forward;
zAxis.Normalize();
Vector3 xAxis = Vector3.Cross(Vector3.Up,zAxis);
xAxis.Normalize();
Vector3 yAxis = Vector3.Cross(xAxis,zAxis);
yAxis.Normalize();
viewMatrix.Forward = zAxis;
viewMatrix.Up = yAxis;
viewMatrix.Right = xAxis;
}
Check out the following question as it has the answer to what you're looking for: Camera Rotation