AVAssetReaderTrackOutput
(added to your AVAssetReader
) will output a CVPixelBufferRef
, of which you can specify your preferred format to upload to OpenGL via glTexImage
or glTexSubImage
.
How do I get an openGL texture out of a video using AVFoundation?
-
13-06-2023 - |
Question
Building a 64bit native OSX (not iOS) ap targeting 10.7+. Somewhat new to dealing with video files in a Cocoa universe.
I would like to be able to open a video file and display the output inside an openGL render (IE, I would like to be able to efficiently access the framebuffer of a video and turn each frame into an opengl texture.)
Conceptually this seems straightforward, but I'm having a difficult time wading through various (old and deprecated)examples and options, all of which seem to have been recently deprecated in favor of AVFoundation. It's possible I missed something, but examples of using AVFoundation with OpenGL seem thin on the ground.
To clarify a bit further, this sample application (QTCoreVideo101 from Apple) does more or less exactly what I want, except that it is built around the deprecated QTKit will therefore not even compile in 64 bit.
I am reading through the AVFoundation docs right now, but I'm still not sure it make sense to try and get a glTexture out of AVFoundation or if I should be looking elsewhere.
UPDATE
This is the solution I ended up going with. "thisLayer.layerSource.videoPlayerOutput" is AVPlayerItemVideoOutput object.
if ([thisLayer.layerSource.videoPlayerOutput hasNewPixelBufferForItemTime:playerTime]){
frameBuffer= [thisLayer.layerSource.videoPlayerOutput copyPixelBufferForItemTime:playerTime itemTimeForDisplay:NULL];
CVReturn result= CVOpenGLTextureCacheCreateTextureFromImage(NULL,
textureCache,
frameBuffer,
NULL,
&textureRef);
if(result == kCVReturnSuccess){
// These appear to be GL_TEXTURE_RECTANGLE_ARB
thisLayer.layerSource.vid_glTextureTarget=CVOpenGLTextureGetTarget(textureRef);
thisLayer.layerSource.vid_glTexture=CVOpenGLTextureGetName(textureRef);
thisLayer.layerSource.vid_glTextureSize=NSMakeSize(CVPixelBufferGetWidth(frameBuffer), CVPixelBufferGetHeight(frameBuffer));
thisLayer.layerSource.vid_ciimage=[CIImage imageWithCVImageBuffer:frameBuffer];
CFRelease(textureRef);
CVOpenGLTextureCacheFlush(textureCache, 0);
}else{
NSLog(@"INTERNAL ERROR FAILED WITH CODE: %i",result);
}
CVBufferRelease(frameBuffer);
}
La solution
Autres conseils
I've been looking at this too, this is my current solution:
- (BOOL) renderWithCVPixelBufferForTime: (NSTimeInterval) time
{
CMTime vTime = [self.playeroutput itemTimeForHostTime:CACurrentMediaTime()];
if ([self.playeroutput hasNewPixelBufferForItemTime:vTime]) {
if (_cvPixelBufferRef) {
CVPixelBufferUnlockBaseAddress(_cvPixelBufferRef, kCVPixelBufferLock_ReadOnly);
CVPixelBufferRelease(_cvPixelBufferRef);
}
_cvPixelBufferRef = [self.playeroutput copyPixelBufferForItemTime:vTime itemTimeForDisplay:NULL];
CVPixelBufferLockBaseAddress(_cvPixelBufferRef, kCVPixelBufferLock_ReadOnly);
GLsizei texWidth = CVPixelBufferGetWidth(_cvPixelBufferRef);
GLsizei texHeight = CVPixelBufferGetHeight(_cvPixelBufferRef);
GLvoid *baseAddress = CVPixelBufferGetBaseAddress(_cvPixelBufferRef);
glBindTexture(GL_TEXTURE_RECTANGLE_ARB, self.textureName);
glTexParameteri(GL_TEXTURE_RECTANGLE_ARB, GL_TEXTURE_STORAGE_HINT_APPLE , GL_STORAGE_CACHED_APPLE);
glTexImage2D(GL_TEXTURE_RECTANGLE_ARB, 0, GL_RGB, texWidth, texHeight, 0, GL_YCBCR_422_APPLE, GL_UNSIGNED_SHORT_8_8_APPLE, baseAddress);
glBindTexture(GL_TEXTURE_RECTANGLE_ARB, 0);
}
return YES;
}
However, I'm wondering if there isn't a more efficient solution, I also have a question on the same topic with several approaches:
Best path from AVPlayerItemVideoOutput to openGL Texture
The lock base address call is a hog, I'm not sure it's truly required.