Modifying rotation matrices is not enough, you need to change position of the camera. In structure from motion problem it is assumed that scene is static, while camera is moving. You can consider such case because only relational movement is important.
Let the extrinsic camera matrix be A = R[I | -C], where C is position of camera center in global frame and R is rotation from global frame to the camera frame. Let Ra represent rotation by angle alpha about vertical axis in global frame. It can be written as (cos(alpha),-sin(alpha),0;sin(alpha),cos(alpha),0;0,0,1). Then the required camera matrix can be computed as A2 = R2[I | -C2], where R2 = R * transpose(Ra) and C2 = Ra * C.
However, you should ensure two things when using this approach. Firstly vertical axis of global frame must correspond to a real-world vertical direction. Secondly the origin of global frame must lie on the axis of the camera center rotation. The latter can be achieved by putting the object at the origin of global frame.
If angles are measured inaccurately or global frame is not centered well, then the computed extrinsic matrix can also be inaccurate. It can be used as an initial estimate for a structure from motion algorithm in this case. The other alternative is to calibrate the camera for each frame, not only the first one.