Your question is mathematical. Given a matrix of 3x3 decide whether it represents a good rigid transformation. It is hard to define what is "good" but here are some clues that can help you
- Homography should preserve the direction of polygonal points. Design a simple test. points (0,0), (imwidth,0), (width,height), (0,height) represent a quadrilateral with clockwise arranged points. Apply homography on those points and see if they are still clockwise arranged if they become counter clockwise your homography is flipping (mirroring) the image which is sometimes still ok. But if your points are out of order than you have a "bad homography"
- The homography doesn't change the scale of the object too much. For example if you expect it to shrink or enlarge the image by a factor of up to X, just check this rule. Transform the 4 points (0,0), (imwidth,0), (width-1,height), (0,height) with homography and calculate the area of the quadrilateral (opencv method of calculating area of polygon) if the ratio of areas is too big (or too small), you probably have an error.
- Good homography is usually uses low values of perspectivity. Typically if the size of the image is ~1000x1000 pixels those values should be ~0.005-0.001. High perspectivity will cause enormous distortions which are probably an error. If you don't know where those values are located read my post: trying to understand the Affine Transform . It explains the affine transform math and the other 2 values are perspective parameters.
I think that if you check the above 3 condition (condition 2 is the most important) you will be able to detect most of the problems. Good luck