I think my answer to your previous question still applies, for the most part.
If you want to compute the area of the region bounded by the smaller angle of the two lines and the boundaries of the square, then you can forget about the intersection, and forget about all the different cases.
You can just use the fact that
- the area of the square S is equal to 4
the value of this integral
A = quadgk(@(x) ... abs( max(min(line1(x),+1),-1) - max(min(line2(x),+1),-1) ), -1, +1);
gives you the area between the lines (sometimes the large angle, sometimes the small angle)
- the value of of
min(A, S-A)
is the correct answer (always the small angle).