One way to grow a line from contour points is to track pixels on the contour while checking for line straightness. At each new pixel you estimate the longest distance from the covered contour and the line connecting contour's beginning and the current contour point. When this distance exceeds a threshold you stop growing a line. In the picture below, the line is red, contour is black and the distance is green.
In your case (since you aren't growing the line) all you need to do is to connect two contour terminals and then scan the contour to get all the distances to the line and select the largest one. Compare it to your threshold to decide on the linearity of the segment.
Here is how to find the distance from point x, y to the line ax+by+c=0 or to the line defined by two points as in your case: link