how to generateRandomTransform with OpenCV

Question 1

You can build a homography matrix by multiplying random scaling, translation and rotation matrices together:

H = T . S . R

The rotation matrix can be generated by using e.g. Euler angles (three parameter rotation).

The translation matrix is an identity matrix with last column entries corresponding to your point translation in each dimension.

Finally, the scale matrix has scaling entries for each dimension on its diagonal.

The final homography matrix H can be normalized by dividing it by its last entry (bottom right corner). You can check whether this entry is not too small (if you need 2D homography and project the transformed points in the same plane).

The advantage of this approach is a better control over your parameters (e.g. the translation corresponds to your typical translation amounts used) and the generated homographies are guaranteed to be invertible as long as the three simple matrices are invertible as well.

Of course, you can generate well-behaving homographies for a unit square, for example, and then scale the matrix to fit your actual problem (e.g. image size in pixels).

Question 2

This post explains how to generate random homographies from scratch - you have control over rotation, translation, shearing and projective distortion:

https://medium.com/uruvideo/dataset-augmentation-with-random-homographies-a8f4b44830d4

To briefly summarize the link above, a homography can be written as the multiplication of a euclidean transformation, an affine/shearing transformation and a projective transformation matrices.

Question 3

Homography also has a shear and projective component. What you described is rather an Affine transform.