How to do "AND" stencil in OpenGL?

Question

I have a scene rendered, and I'm doing some post-processing that takes place only on a portion of the screen. I use a sphere to "cut" the area, but I get the "full" sphere, whereas I would need "AND" cut of the surfaces that are inside the sphere. It's a bit difficult to explain, but hopefully the two following mock-ups help. Let's say the scene is the blue thing, and the green sphere is a stencil mesh. The right picture shows the resulting stencil.

So... How can I do this? At the moment I do something like this:

GL.Enable(EnableCap.DepthTest);
GL.ColorMask(false, false, false, false);
GL.CullFace(CullFaceMode.Front);

GL.Enable(EnableCap.StencilTest);
GL.Clear(ClearBufferMask.StencilBufferBit);

GL.StencilFunc(StencilFunction.Always, 1, 1);
GL.StencilOp(StencilOp.Incr, StencilOp.Incr, StencilOp.Incr);

Resources.R.StencilMesh.Render();

GL.StencilFunc(StencilFunction.Equal, 1, 1);
GL.StencilOp(StencilOp.Keep, StencilOp.Keep, StencilOp.Keep);

This, however results in a stencil buffer with a full sphere, whereas I want only the surfaces that are left inside the sphere.

Answer 1

If I'm understanding you correctly, than I think you've got the wrong parameters in your first call to glStencilOp.

The three operations are respectively:

1. What to do when the stencil test fails
2. What to do when the stencil test passes, and the depth test fails
3. What to do when the stencil and depth test pass.

Since your stencil test always passes, we can ignore #1, but if you only want to draw in the stencil buffer the place where the sphere would be drawn (and not filled in the area occluded by the depth buffer), you should not increment the stencil buffer in case #2.

So I would instead try:

GL.StencilFunc(StencilFunction.Always, 1, 1);
GL.StencilOp(StencilOp.Keep, StencilOp.Keep, StencilOp.Incr);
Resources.R.StencilMesh.Render();