How to set blend function on opengl for two overlapping objects

Question

Hi I needed to draw a round corner rectangle.

I followed the procedure of the above image. I first drew the green rectangle. Then I drew the two black rectangles.And then I drew circles on the edges to make the corner round. Now what I get after doing this is in the image below.

As it can be seen that the corner circles have less transparency on the portions where they overlap with the rectangles. But more transparency when not overlapped with the rectangles. The rectangles have alpha set to 0.5f. and the circle also have 0.5f alpha. So thats why its white on the overlapped portions and transparent on non overlapped portions. I want the overlapped portions to have same transparency as the rectangle so that the overlapped circle portion can not be seen.My blend function is glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA); I tried to understand the blend functions in more details in here. But I could not understand anything. My code is below,

glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glViewport(0, 0, (int) screenWidth, (int) screenHeight);
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
glOrthof(0.0f, (double)screenWidth / screenHeight, 0.0f, 1.0f, -1.0f, 1.0f);

glEnable(GL_TEXTURE_2D);
glEnableClientState(GL_VERTEX_ARRAY);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);

glEnable(GL_COLOR_MATERIAL);
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glDisable(GL_TEXTURE_2D);
glBlendFunc(GL_ONE_MINUS_DST_ALPHA, GL_DST_ALPHA);
glEnableClientState(GL_COLOR_ARRAY);

glVertexPointer(3, GL_FLOAT, 0, bubbleMiddleRectStartCoord);
glColorPointer(4, GL_FLOAT, 0, rectColor);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

glVertexPointer(3, GL_FLOAT, 0, bubbleTopRectStartCoord);
glColorPointer(4, GL_FLOAT, 0, rectColor);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

glVertexPointer(3, GL_FLOAT, 0, bubbleBottomRectStartCoord);
glColorPointer(4, GL_FLOAT, 0, rectColor);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

//smooth edge of the bubble rectangle
drawCircle(triangleAmount,bubbleEdgeRadius,bubbleMiddleRectStartCoord->upperLeft.x+bubbleEdgeRadius,bubbleMiddleRectStartCoord->upperLeft.y,255,255,255,128);
drawCircle(triangleAmount,bubbleEdgeRadius,bubbleMiddleRectStartCoord->lowerLeft.x+bubbleEdgeRadius,bubbleMiddleRectStartCoord->lowerLeft.y,255,255,255,128);
drawCircle(triangleAmount,bubbleEdgeRadius,bubbleMiddleRectStartCoord->upperRight.x-bubbleEdgeRadius,bubbleMiddleRectStartCoord->upperRight.y,255,255,255,128);
drawCircle(triangleAmount,bubbleEdgeRadius,bubbleMiddleRectStartCoord->lowerRight.x-bubbleEdgeRadius,bubbleMiddleRectStartCoord->lowerRight.y,255,255,255,128);

glDisableClientState(GL_COLOR_ARRAY);
glEnable(GL_TEXTURE_2D);
glDisable(GL_BLEND);
glDisableClientState(GL_VERTEX_ARRAY);
glDisableClientState(GL_TEXTURE_COORD_ARRAY);

glDisableClientState(GL_COLOR_MATERIAL);
glDisable(GL_TEXTURE_2D);
swapBuffers();

rectColor has value

GLfloat rectColor[]=
{
    1.0f,1.0f,1.0f,0.5,
    1.0f,1.0f,1.0f,0.5,
    1.0f,1.0f,1.0f,0.5,
    1.0f,1.0f,1.0f,0.5
};

drawCircle function generates the points for the circle and draws it. The drawing portion of that function is

glVertexPointer(2, GL_FLOAT, 0, vertices);
 glColorPointer(4, GL_UNSIGNED_BYTE, 0, color);
 glDrawArrays(GL_TRIANGLE_FAN, 0, triangleAmount+2);

Can anyone help me to solve the problem? Thanks.

EDIT: this is how it looks after using those two blend functions.

Answer 1

I see where you are going with this and seeing your result you probably only need to disable blend while you are drawing the mask (the 3 rectangles and 4 circles), then using glBlendFunc(GL_DST_ALPHA, GL_ZERO). Though this will only work if nothing has already been drawn on the scene.

To explain what you did there is you are drawing a white color with .5 alpha and blending it. Consider on the beginning the pixel color "destination" is (0,0,0,0) and incoming "source" is always in your case (1,1,1,.5). Lets say source color is "S" and destination is "D" while the components are (r,g,b,a) so that source alpha is "S.a" what you wrote in your blend function is:

output = S*S.a + D*(1.0-S.a) = 
(1,1,1,.5)*.5 + (0,0,0,0)*(1.0-.5) = 
(.5, .5, .5, .25) + (0,0,0,0) =
(.5, .5, .5, .25)

so when you draw your circle over the already drawn rectangle:

output = S*S.a + D*(1.0-S.a) = 
(1,1,1,.5)*.5 + (.5, .5, .5, .25)*(1.0-.5) = 
(.5, .5, .5, .25) + (.25, .25, .25, .125) =
(.75, .75, .75, .375)

resulting in alpha difference. So from this I hope you can understand what the 2 parameters mean in the blend function: First one tells what factor to use to multiply the source (incoming) color and the second one how to multiply the destination color. In the end they are summed together.

So for your case you would like to force the alpha channel to some value everywhere you draw those primitives. To achieve that you would need S*1.0 + D*.0 and parameters for that are glBlendFunc(GL_ONE, GL_ZERO), though this is the same as just disabling the blend. Only writing this primitives would produce a white(gray) rounded rect with transparency of .5 while all the rest is fully transparent. Now after this you need to set blend function to multiply your incoming color with the destination alpha glBlendFunc(GL_DST_ALPHA, GL_ZERO).

EDIT:

I did not totally understand what you want to achieve till now. As I mentioned above, this will not work if you already have some scene drawn.

To overlay an existing scene with some complex object (in this case the object is overlapping itself on some parts) it would be most bulletproof to use a stencil buffer. Creating it is much like depth buffer but you may consider it as another color channel, it is easy to draw to it and later use it so you might want to look at it at some point.

In your case it is probably safe to say this is your main buffer and is meant for displaying. In that case you can just use the alpha channel:

To draw only to alpha channel you have to set glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_TRUE) and when you are done, put all the parameters to true.

1. To clear the alpha channel you have to draw a fullscreen rect with some color with desired alpha (I suggest you use (1,1,1,1)) and draw only to alpha channel
2. To draw that mask (the 3 rects and 4 circles) use glBlendFunc(GL_ONE, GL_ZERO) and color (1,1,1, 1-desiredAlpha)
3. To draw your rounded label use glBlendFunc(GL_ONE_MINUS_DST_ALPHA, GL_DST_ALPHA)

So the procedure would be:

//your background is drawn, time to overly labels
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_TRUE);
glColor(1.0f, 1.0f, 1.0f, 1.0f);
//draw fullscreen rect
glBlendFunc(GL_ONE, GL_ZERO);
glColor(1.0f, 1.0f, 1.0f, 1.0f-.5f);
//draw 3 rects and 4 circles
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glBlendFunc(GL_ONE_MINUS_DST_ALPHA, GL_DST_ALPHA);
//draw the label as a normal rect (the rounded parts will be trimmed because of alpha channel)

and you can just repeat that in a for loop for all the labels.

I know things got a bit complicated but what you are trying to do is not as easy as it would seem. I presented this solution to you because this way you have least code to change, in general I would suggest to use stencil buffer (already mentioned) or a FBO (frame buffer object). The FBO system would be to create another frame buffer and attach a texture to it, draw the whole label object to it and then use the bound texture to draw it to main screen.