Cocoa OpengGL: draw texture and round corners

Question

If I use a standard NSWindow to host a NSOpenGLView which extends for all the window frame, the two bottom corners of the window are automatically rounded.

When swithcing to NSBorderlessWindowMask I have to handle corner rounding myself. I have already implemented a transparent custom NSWindow and a rounded custom NSView and they both work fine. After that I have implemented a transparent NSOpenGLContext by setting NSOpenGLCPSurfaceOpacity to 0. If I set a color background for the OpenGL context, the view is drawn correctly and I obtain the desired rounded corner.

But, since the app is a movie player, I need to draw the texture corresponding to every movie frame.

When I do this (using glTexCoord2f and glVertex2f) the texture is drawn till corners and therefore the image is drawn till outside of the rounded corners and I loose the rounded aspect of my window.

What does the system do when the window is standard and non NSBorderlessWindowMask that I can't seem to be able to reproduce?

What is the best way to round the corner of the texture while drawing it to the frame buffer?

Answer 1

You could apply the texture to a geometry with rounded corners, use an additional alpha mask on the movie texture with rounded corners, or use the stencil test to round off your viewport's corners.