opengl transparency order algorithm

Question

Is there a simple algorithm, given a depth vector (ie the one going into the screen) to determine which polygon (or rather, just which triangle) is 'behind' the other one from their coordinates so I can sort them and get their transparency effects to behave appropriately? I was thinking about comparing overlaps of minimum and maximum values but this can fail, particularly when the range of one polygon in the depth direction goes beyond both the minimum and maximum depth direction of the other. Any ideas?

Answer 1

After investigation it would appear NO there is no such way. See opengl.org/wiki/Transparency_Sorting. Basically the problem is ill defined since we are talking about 2d overlaps in 3d space so can get pathological cases where A>B B>C C>A: here there is no possible order which works.

The problem is that the alpha blending function

glBlendEquation(GL_FUNC_ADD);
glBendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);

is not commutative: it relies on subtraction. This is why order matters: A-B != B-A. This however is the best way to do transparency which lots of complicated color effects etc. The way round this is either order independent transparency (eg depth peeling which is apprently costly) or brute force eg. divide up your polygons into arbitrary smaller ones until the sort can be done. This could be made more sophisticated by only dividing up your polygons along the lines where they overlap in the x-y dimensions of the view port. But it's still all a bit of a pain.

If on the other hand your transparency is a simple example (ie. when it explicitly doesn't make a difference to the visuals what order the transparencies are in eg. for me all transparent objects were rectangles of the same grey colour ) none of this matters! It's all a red herring: the problems comes from THINKING it does matter. What we need to do is stop providing information to the z-buffer and telling opengl that it needs to consider the order (which then subsequently gets messed up). So in this case we just need to write

glBlendFunc(GL_SRC_ALPHA,GL_ONE_MINUS_SRC_ALPHA);   
glEnable( GL_BLEND );
glEnable(GL_DEPTH_TEST);
glDepthMask(GL_FALSE);
glBegin(GL_TRIANGLES);

// draw here
glEnd();
glDisable(GL_BLEND);
glDisable(GL_DEPTH_TEST);
glDepthMask(GL_TRUE);

where the glDepthMask(GL_FALSE) is ensuring depth information isn't being written. You can rotate all you want and the transparencies look fine! But this is dependent on being in a situation where the image outcome DOESN'T depend on the order: it's just that with all the documentation you might think your situation does depend on it when it might not.

If it does depend on the order (it would look different from one angle as opposed to another using the above and you don't want it to) and you don't want to do fancy things or complicated sorts you can use a 'lesser' form of transparency by using commutative blending (multiplicative or additive). You still have to disable z writing and you can do things like

Multiplicative

glBlendFunc(GL_ZERO,GL_SRC_COLOR);   
glEnable( GL_BLEND );
glEnable(GL_DEPTH_TEST);
glDepthMask(GL_FALSE);
glBegin(GL_TRIANGLES);

// draw here
glEnd();
glDisable(GL_BLEND);
glDisable(GL_DEPTH_TEST);
glDepthMask(GL_TRUE);

Additive

glBlendFunc(GL_ONE,GL_ONE);   
glEnable( GL_BLEND );
glEnable(GL_DEPTH_TEST);
glDepthMask(GL_FALSE);
glBegin(GL_TRIANGLES);

// draw here
glEnd();
glDisable(GL_BLEND);
glDisable(GL_DEPTH_TEST);
glDepthMask(GL_TRUE);

and play about with these to see if they look OK. The point is try these 'lesser' approaches to transparency before looking into complicated sorts/order independent approaches because these will have much lower overheads and might work for your situation depending on what you are doing.