The significance of the rotation matrix after computing structure from motion

Question

I've fully implemented the algorithm and I'm a bit confused by how the rotation matrix works. So you end with a "structure" matrix which is 3xP, and the contents (if I'm correct) are P 3D points (so rows are x,y,z).

The rotation matrix however is 2fx3. F being the number of frames since initially we stack 3 frames of tracked feature points into a matrix. And it's 2f because the top half are the x coordinates and the bottom half the y coordinates.

Anyway, the resulting matrix is this 2fx3 and it seems like you have 2 rotation matrices so I'm a bit confused how it corresponds to a normal rotation matrix

Here's a short overview of the algorithm http://www.cs.huji.ac.il/~csip/sfm.pdf

Answer 1

I actually figured out the answer. So like I said the R matrix is of the size 2fx3 and I was confused how that corresponded to a normal 3x3 rotation matrix. So it turns out that since R is stacked such that you have

r1x
r2x
r3x
r1y
r2y
r3y

Where each row is a 1x3 vector that corresponds to a row in a normal rotation matrix to get the rotation from the initial points to the new ones you take the corresponding r rows for x,y and cross them for z. So to get the rotation matrix for the 1st frame it would be (each of these is a 1x3 vector)

r1x
r1y
cross(r1x, r1y)