Assuming you're looking for red lines...
- Threshold just the red component of the image
- Compute hough lines and look for line segments of an estimated length (if you know the length of the sides of the triangle/square you're looking for).
- Once you have this list, find combinations of lines that form triangles and squares.
- Verify each candidate triangle/square by checking that their areas are within expected ranges.
If you follow this method, it is likely that you will find multiple shapes within close proximity of each other i.e. the same triangle/square in the real world will be found multiple times by the algorithm depending on the thickness of the lines. In this case, cluster them by distance and only retain one shape per cluster.
Another option is
- Threshold the red component of the image.
- Find contours.
- Check for closed contours.
- For every close contour, check if the shape resembles an equilateral triangle or a square by plotting histograms of slopes of individual points on the contour. The histogram for a square will have two highly populated bins, while that of a triangle will have three highly populated bins.