So it turns out it was not the code I had posted that caused the problem. Thanks to some help in the comment, I was able to find it was this code when I'm determining the nearest object to the camera:
float nearest_t = FAR_CLIP;
int nearest_index = 0;
for (int j=0; j<NumObjects; j++)
{
float t = FAR_CLIP;
t = FindRayObjectIntersection(r, objects[j]);
if (t < nearest_t && t > EPSILON && t < FAR_CLIP)
{
nearest_t = t;
nearest_index = j;
}
}
When determining t, sometimes the triangles were so close together that the t < nearest_t
had an almost probabilistic result, since the intersections were roughly the same distance from the camera.
My initial solution was to change the inner if-statement to:
if (t < nearest_t-EPSILON && t > EPSILON && t < FAR_CLIP)
This ensures that if two intersections are very close together, it will always choose the first object to display (unless the second object is closer by at least EPSILON
). Here is a resulting image (with reflections disabled):
Now there were still some small artifacts, so it was clear that there was still a slight problem. So upon some discussion in the comments, @Soonts came up with the idea of blending the triangles' colours. This lead me to have to change the above code further in order to keep track of the distance to both triangles:
if (t > EPSILON && t < FAR_CLIP && abs(nearest_t - t) < EPSILON)
{
nearest_index2 = nearest_index;
nearest_t2 = nearest_t;
}
if (t < nearest_t+EPSILON && t > EPSILON && t < FAR_CLIP)
{
nearest_t = t;
nearest_index = j;
}
I also added this colour blending code:
OverallColor = mix(c1, c2, 0.5f * abs(T1 - T2) / EPSILON);
After these two steps, and honestly I think this effect was more from the logic change than the blending change, I got this result:
I hope others find this solution helpful, or at the very least that it sparks some ideas to solve your own problems. As a final example, here is the beautiful result with reflections, softer shadows and some anti-aliasing all turned on:
Happy raytracing!