How do I get an openGL texture out of a video using AVFoundation?

Question

Building a 64bit native OSX (not iOS) ap targeting 10.7+. Somewhat new to dealing with video files in a Cocoa universe.

I would like to be able to open a video file and display the output inside an openGL render (IE, I would like to be able to efficiently access the framebuffer of a video and turn each frame into an opengl texture.)

Conceptually this seems straightforward, but I'm having a difficult time wading through various (old and deprecated)examples and options, all of which seem to have been recently deprecated in favor of AVFoundation. It's possible I missed something, but examples of using AVFoundation with OpenGL seem thin on the ground.

To clarify a bit further, this sample application (QTCoreVideo101 from Apple) does more or less exactly what I want, except that it is built around the deprecated QTKit will therefore not even compile in 64 bit.

I am reading through the AVFoundation docs right now, but I'm still not sure it make sense to try and get a glTexture out of AVFoundation or if I should be looking elsewhere.

UPDATE

This is the solution I ended up going with. "thisLayer.layerSource.videoPlayerOutput" is AVPlayerItemVideoOutput object.

if ([thisLayer.layerSource.videoPlayerOutput hasNewPixelBufferForItemTime:playerTime]){

    frameBuffer= [thisLayer.layerSource.videoPlayerOutput copyPixelBufferForItemTime:playerTime itemTimeForDisplay:NULL];

    CVReturn result= CVOpenGLTextureCacheCreateTextureFromImage(NULL,
                                           textureCache,
                                           frameBuffer,
                                           NULL,
                                           &textureRef);
    if(result == kCVReturnSuccess){
            // These appear to be GL_TEXTURE_RECTANGLE_ARB
            thisLayer.layerSource.vid_glTextureTarget=CVOpenGLTextureGetTarget(textureRef);
            thisLayer.layerSource.vid_glTexture=CVOpenGLTextureGetName(textureRef);
            thisLayer.layerSource.vid_glTextureSize=NSMakeSize(CVPixelBufferGetWidth(frameBuffer), CVPixelBufferGetHeight(frameBuffer));
            thisLayer.layerSource.vid_ciimage=[CIImage imageWithCVImageBuffer:frameBuffer];
            CFRelease(textureRef);
            CVOpenGLTextureCacheFlush(textureCache, 0);
    }else{
            NSLog(@"INTERNAL ERROR FAILED WITH CODE: %i",result);
    }
    CVBufferRelease(frameBuffer);
}

Answer 1

AVAssetReaderTrackOutput (added to your AVAssetReader) will output a CVPixelBufferRef, of which you can specify your preferred format to upload to OpenGL via glTexImage or glTexSubImage.

Answer 2

I've been looking at this too, this is my current solution:

- (BOOL) renderWithCVPixelBufferForTime: (NSTimeInterval) time
{
    CMTime vTime = [self.playeroutput itemTimeForHostTime:CACurrentMediaTime()];

    if ([self.playeroutput hasNewPixelBufferForItemTime:vTime]) {
        if (_cvPixelBufferRef) {
            CVPixelBufferUnlockBaseAddress(_cvPixelBufferRef, kCVPixelBufferLock_ReadOnly);
            CVPixelBufferRelease(_cvPixelBufferRef);
        }
        _cvPixelBufferRef = [self.playeroutput copyPixelBufferForItemTime:vTime itemTimeForDisplay:NULL];

        CVPixelBufferLockBaseAddress(_cvPixelBufferRef, kCVPixelBufferLock_ReadOnly);
        GLsizei       texWidth    = CVPixelBufferGetWidth(_cvPixelBufferRef);
        GLsizei       texHeight   = CVPixelBufferGetHeight(_cvPixelBufferRef);
        GLvoid *baseAddress = CVPixelBufferGetBaseAddress(_cvPixelBufferRef);


        glBindTexture(GL_TEXTURE_RECTANGLE_ARB, self.textureName);
        glTexParameteri(GL_TEXTURE_RECTANGLE_ARB, GL_TEXTURE_STORAGE_HINT_APPLE , GL_STORAGE_CACHED_APPLE);
        glTexImage2D(GL_TEXTURE_RECTANGLE_ARB, 0, GL_RGB, texWidth, texHeight, 0, GL_YCBCR_422_APPLE, GL_UNSIGNED_SHORT_8_8_APPLE, baseAddress);

        glBindTexture(GL_TEXTURE_RECTANGLE_ARB, 0);
    } 
    return YES;
} 

However, I'm wondering if there isn't a more efficient solution, I also have a question on the same topic with several approaches:

Best path from AVPlayerItemVideoOutput to openGL Texture

The lock base address call is a hog, I'm not sure it's truly required.