texture for YUV420 to RGB conversion in OpenGL ES

Question

I have to convert and display YUV420P images to RGB colorspace using the AMD GPU on a Freescale iMX53 processor (OpenGL ES 2.0, EGL). Linux OS, no X11. To achieve this I should be able to create an appropriate image holding the YUV420P data: this could be either a YUV420P/YV12 image type or 3 simple 8-bit images, one for each component (Y, U, V).

glTexImage2D is excluded, because it's slow, the YUV420P frames are the results of a real time video decoding @25FPS and with glTexImage2D we can't keep the desired framerate.

There's an alternative: eglCreateImageKHR/glEGLImageTargetTexture2DOES. The only problem is that these can't handle any image format that would be suitable for YUV420/YV12 data.

EGLint attribs[] = {
  EGL_WIDTH, 800,
  EGL_HEIGHT, 480,
  EGL_IMAGE_FORMAT_FSL, EGL_FORMAT_YUV_YV12_FSL,
  EGL_NONE
};

EGLint const req_attribs[] = {
  EGL_RED_SIZE, 5,
  EGL_GREEN_SIZE, 6,
  EGL_BLUE_SIZE, 5,
  EGL_ALPHA_SIZE, 0,
  EGL_SAMPLES, 0,
  EGL_COLOR_BUFFER_TYPE, EGL_RGB_BUFFER,
  EGL_SURFACE_TYPE, EGL_WINDOW_BIT,
  EGL_NONE
};

...

display = eglGetDisplay(EGL_DEFAULT_DISPLAY);
eglInitialize(display, NULL, NULL);
eglBindAPI(EGL_OPENGL_ES_API);
eglChooseConfig(display, req_attribs, config, ARRAY_SIZE(config), &num_configs);
ctx = eglCreateContext(display, curr_config, NULL, NULL);
surface = eglCreateWindowSurface(display, curr_config, fb_handle, NULL);

...

EGLImageKHR yuv_img = eglCreateImageKHR(display, EGL_NO_CONTEXT, EGL_NEW_IMAGE_FSL, NULL, attribs); 
eglQueryImageFSL(display, yuv_img, EGL_CLIENTBUFFER_TYPE_FSL, (EGLint *)&ptr);
glGenTextures(1, &texture);
glBindTexture(GL_TEXTURE_2D, texture);
glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, yuv_img);

glEGLImageTargetTexture2DOES(...) fails. If I change the appropriate line in 'attribs' to this:

EGL_IMAGE_FORMAT_FSL, EGL_FORMAT_RGB_565_FSL,

then the image can be assigned to an OpenGL ES texture, but it's not appropriate to hold either 8-bit data (Y/U/V) or a YUV420/YV12 data. Searching the net (including Freescale community forum) I've haven't found any solution to this.

How can I create an image which:

• is fast to create;
• eventually can be assigned to an already existing buffer (physical address or virtual address is given);
• can be used in the fragment/vertex shader program to perform YUV --> RGB conversion;

Constraint is to avoid unneccessary memcpy(...)s due to performance reasons.

Answer 1

I have implemented this on the i.MX53 for several YUV formats and it works really well. I have a published article about it, although it was generalized to cover more Android platforms:

http://software.intel.com/en-us/articles/using-opengl-es-to-accelerate-apps-with-legacy-2d-guis

I suspect your problem is that you are not binding to the correct texture target. It should be like this:

glEGLImageTargetTexture2DOES(GL_TEXTURE_EXTERNAL_OES, hEglImage[iTextureIndex]);

glBindTexture(GL_TEXTURE_EXTERNAL_OES, hTexture[iIndex]);   

And the eglImageAttributes should be one of these:

EGLint eglImageAttributes[] = {EGL_WIDTH, iTextureWidth, EGL_HEIGHT, iTextureHeight, EGL_IMAGE_FORMAT_FSL, EGL_FORMAT_YUV_YV12_FSL, EGL_NONE};
EGLint eglImageAttributes[] = {EGL_WIDTH, iTextureWidth, EGL_HEIGHT, iTextureHeight, EGL_IMAGE_FORMAT_FSL, EGL_FORMAT_YUV_NV21_FSL, EGL_NONE};
EGLint eglImageAttributes[] = {EGL_WIDTH, iTextureWidth, EGL_HEIGHT, iTextureHeight, EGL_IMAGE_FORMAT_FSL, EGL_FORMAT_YUV_UYVY_FSL, EGL_NONE};

hEglImage[iTextureIndex] = eglCreateImageKHR(eglDisplay, EGL_NO_CONTEXT, EGL_NEW_IMAGE_FSL, NULL, eglImageAttributes);

struct EGLImageInfoFSL EglImageInfo;
eglQueryImageFSL(eglDisplay, hEglImage[iTextureIndex], EGL_CLIENTBUFFER_TYPE_FSL, (EGLint *)&EglImageInfo);

Although this feature of the Freescale i.MX53 platform makes YUV to RGB color space conversion for video extremely fast, it does have a couple of limitations:

1. It only supports those 3 YUV formats.
2. eglCreateImageKHR() must allocate the buffers. There is no way to make it use existing buffers. Freescale confirmed that the NULL pointer can not be anything else, which technically violates the Khronos specification.

Freescale has resolved these problems on the i.MX6 platform, although the architecture is really different. Hope this helps.