One approach for doing this can be 1. Extract geometric markers (special points), for both input and output, for example by using the Harris corner detector. 2. Compare every possible pair of triangles on each image . If they have the same angles, they define a transformation (resize, rotate,shift, and mirror). Google "geometric hashing" 3 Count how many points in the target agree with the transformed source, and you have a score for the match. 4. Verify the best match by other means (histogram, edge-registration).
This approach is slow in theory (o(n^6) in the number of points) but I believe it can be practical using an appropriate choice of n and some tricks to speed it up.