Opengl Stencil Buffer set when not transparent

Question

I was trying to configure my stencil buffer so that, when enabled, it would set when the pixel drawn is not transparent (thus creating a map of pixels that light can collide with). What I've done is:

    glClearStencil(0); //clear stencil
    glStencilFunc(GL_EQUAL, 0xFF, 0x000000FF); //only where alpha (mask : 0x000000FF) is 0xFF (opaque)
    glStencilOp(GL_INCR, GL_KEEP, GL_KEEP); //increment if passes (if it is opaque)

    render(); //withing this method I sometimes disable the whole thing to draw the floor, for example

Then, i use the following code to test:

    /* TURN OFF STENCIL */
    glEnable(GL_STENCIL_TEST); //re-enable
    glStencilFunc(GL_EQUAL, 0, 1); //if the test is equal to 1
    glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); //do not change stencil buffer

    ImageInfo.drawColorSquare(0, 0, Configurations.SCREEN_WIDTH, Configurations.GAME_HEIGHT, Color.BLUE); //drwa blue square

    glDisable(GL_STENCIL_TEST); //disable

However, there are two problems:

1. It doesn't seem to be ignoring transparent pixels, as it should;
2. If a region overlaps with another, then it reverses - for example, it sets to one, then another region is drawn in the same area, and it reset it to 0 again.

I don't know why that is happening. Probably something wrong with my mask, I guess - I wasn't absolutely sure how many pixels OpneGL used in the Color Buffer. Also, GL_INCR should add up to the max, and not go back, according to the documentation. Since my stencil buffer size is one bit, it should set to one, try to increase again, fail, and keep on one (instead of reseting).

Answer 1

The Stencil Test is independent of what happens in the color buffer. Setting glStencilFunc ( http://www.opengl.org/sdk/docs/man/xhtml/glStencilFunc.xml ) you can specify how the Stencil Test interacts with what is already stored in the Stencil Buffer.
Setting glStencilOp ( http://www.opengl.org/sdk/docs/man/xhtml/glStencilOp.xml ) gives you the possibility of using the result of the depth test to perform the Stencil Test. A good tutorial that explain the Stencil Test and a very instructive algorithm based upon it can be found here http://ogldev.atspace.co.uk/www/tutorial37/tutorial37.html

Answer 2

The stencil buffer is usually an 8-bit buffer that reserves a small portion of the memory normally used for the depth buffer and is used for advanced rejection of fragments; things like masking to arbitrary shapes rather than using rectangular scissor boxes. It has nothing to do with your color buffer, and to make sure that fragments that have a specific alpha value do not affect the pixels on screen you would use something called an alpha test.

In core OpenGL 3, the fixed-function alpha test is no longer supported, so you would have to implement it in a fragment shader and then discard if it failed to meet your condition.