Draw YUV image using GLKit openGL - iOS

Question

I want to render a yuv image on an iOS device. I presume it can be achieved using openGL. (Actually I have to render multiple such images in succession)

What I understand is that GLKit is an abstraction that iOS created, in which there is a GLKView which will have and handle the render buffer. I am currently trying to use a GLKViewController and frame update is being done successfully with desired fps. This I conform by using glClear function call.

Now the task is to render an image on the view.

There is a class GLKBaseEffect which will have basic shaders. I can't figure out what properties to set, so I just create it and call prepareToDraw before each render.

There is a class for handling textures, GLKTextureLoader, but it appears to me that it only works with Quartz images, i.e., yuv420 planar can't be loaded into the texture using this class.

// create the texture
GLuint texture;
glGenTextures(1, &texture);
glBindTexture(GL_TEXTURE_2D, texture);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8,  width, height, 0, GL_RGBA, GL_UNSIGNED_BYTE, [imageNSData bytes]);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, texture, 0);  

I use this code for generating a texture and binding it, but I don't really know what I am trying to do here. And whatever it is, it doesn't bring up any image on screen, and I don't know what to do next.

I have not created any shaders, assuming baseEffect will have something.

And this https://developer.apple.com/library/ios/documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/WorkingwithEAGLContexts/WorkingwithEAGLContexts.html#//apple_ref/doc/uid/TP40008793-CH103-SW8 tells me that I'll have to use EAGLayer to render images on screen.

Can I use GLKit to render images? If YES, do we have any sample code or tutorial that wouldn't use GLKTextureLoader class (I couldn't find any)? If NO, is there a similar tutorial for render using EAGLLayer (I have not explored about it till now) ?

Answer 1

It sounds like you're really asking about a few different topics here:

• How to draw with GLKView vs CAEAGLLayer
• How GLKBaseEffect and GLKTextureLoader fit into OpenGL ES drawing in general
• How to draw a texture once you have one
• How to render YUV image data

I'll try to address each in turn...

---

GLKView is just fine for any OpenGL ES drawing you want to do -- it does everything that the older documentation you linked to does (setting up framebuffers, CAEAGLLayers, etc) for you so you don't have to write that code. Inside the GLKView drawing method (drawRect: or glkView:drawInRect:, depending on whether you're drawing in a subclass or delegate), you write the same OpenGL ES drawing code you would for CAEAGLLayer (or any other system).

---

You can use GLKView, GLKTextureLoader, and GLKBaseEffect independently of each other. If you want to write all your own drawing code and use your own shaders, you can draw in a GLKView without using GLKBaseEffect. (You can even mix and match GLKBaseEffect and your own stuff, like you see when you create a new Xcode project with the OpenGL Game template.) Likewise, GLKTextureLoader loads image data and spits out the name you'll need for binding it for drawing, and you can use that regardless of whether you're drawing it with GLKBaseEffect.

---

Once you get a texture, whether via GLKTextureLoader or reading/decoding the data yourself and providing it to glTexImage2D, there are three basic steps to drawing with it:

1. Bind the texture name with glBindTexture.
2. Draw some geometry to be textured (using glDrawArrays, glDrawElements, or similar)
3. Have a fragment shader that looks up texels and outputs colors.

If you just want to draw an image that fills your view, just draw a quad (two triangles). Here's the code I use to set up a vertex array object with one quad when I want to draw fullscreen:

typedef struct __attribute__((packed)) {
    GLKVector3 position;
    GLKVector2 texcoord;
} Vertex;

static Vertex vertexData[] = {
    {{-1.0f,  1.0f, -1.0f}, {0.0f,  0.0f}},
    {{-1.0f, -1.0f, -1.0f}, {0.0f,  1.0f}},
    {{ 1.0f,  1.0f, -1.0f}, {1.0f,  0.0f}},
    {{ 1.0f, -1.0f, -1.0f}, {1.0f,  1.0f}},
};

glGenVertexArraysOES(1, &_vertexArray);
glBindVertexArrayOES(_vertexArray);

glGenBuffers(1, &_vertexBuffer);
glBindBuffer(GL_ARRAY_BUFFER, _vertexBuffer);
glBufferData(GL_ARRAY_BUFFER, sizeof(vertexData), vertexData, GL_STATIC_DRAW);

glEnableVertexAttribArray(GLKVertexAttribPosition);
glVertexAttribPointer(GLKVertexAttribPosition, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), (GLvoid *)offsetof(Vertex, position));
glEnableVertexAttribArray(GLKVertexAttribTexCoord0);
glVertexAttribPointer(GLKVertexAttribTexCoord0, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), (GLvoid *)offsetof(Vertex, texcoord));
glBindVertexArrayOES(0);

Then, to draw it:

glBindVertexArrayOES(_vertexArray);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

That's just the geometry-drawing part. Combine this with a GLKBaseEffect -- whose transform property is the default identity transform, and whose texture2d0 property is set up with the name of a texture you've loaded via GLKTextureLoader or other means -- and you'll get a view-filling billboard with your texture on it.

---

Finally, the YUV part... for which I'll mostly punt. Where are you getting your YUV texture data? If it's from the device camera, you should look into CVOpenGLESTexture/CVOpenGLESTextureCache, as covered by this answer. Regardless, you should be able to handle YUV textures using the APPLE_rgb_422 extension, as covered by this answer. You can also look into this answer for some help writing fragment shaders to process YUV to RGB on the fly.