Using GPUImage with AVVideoCompositing

Question

I am trying to use a combination of GPUImage and AVVideoCompositing to implement a live color key filter between two videos. Doing this naively using CIImage imageFromCVPixelBuffer in CGImage in GPUImage from CGImage to CIImage to CVPixelBuffer extremely inefficient and leads to memory problems.

I noticed that there are texture objects, rendering objects, and frame buffers in the GPUImage Framework. I was hoping I could use CVOpenGLESTextureCacheCreateTextureFromImage on iOS to save everything on the GPU.

I don’t think I fully understand the internal workings of the framework because I suggested that I could create a filter chain for the GPUImageTextureInput object and then get the renderTarget filter, which is CVPixelBufferRef . renderTarget below is always nil, and calling imageFromCurrentFrameBuffer will give me a gray square, which is not my image.

Please note that the example below is not a color key, but a simple brightness filter on a single video to try to prove the concept.

 @implementation MyCustomCompositor : NSObject <AVVideoCompositing> - (instancetype)init { self = [super init]; if (self) { CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, [GPUImageContext sharedImageProcessingContext].context, NULL, &_textureCache); } return self; } - (NSDictionary<NSString *,id> *)requiredPixelBufferAttributesForRenderContext { return @{(NSString *)kCVPixelBufferPixelFormatTypeKey : @[@(kCVPixelFormatType_32BGRA)], (NSString *)kCVPixelBufferOpenGLCompatibilityKey : @YES}; } - (NSDictionary<NSString *,id> *)sourcePixelBufferAttributes { return @{(NSString *)kCVPixelBufferPixelFormatTypeKey : @[@(kCVPixelFormatType_32BGRA)], (NSString *)kCVPixelBufferOpenGLCompatibilityKey : @YES}; } - (void)startVideoCompositionRequest:(AVAsynchronousVideoCompositionRequest *)asyncVideoCompositionRequest { @autoreleasepool { CVPixelBufferRef mePixelBuffer = [asyncVideoCompositionRequest sourceFrameByTrackID:200]; CVPixelBufferLockBaseAddress(mePixelBuffer, kCVPixelBufferLock_ReadOnly); CVOpenGLESTextureRef meTextureRef = NULL; size_t width = CVPixelBufferGetWidth(mePixelBuffer); size_t height = CVPixelBufferGetHeight(mePixelBuffer); CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, _textureCache, mePixelBuffer, NULL, GL_TEXTURE_2D, GL_BGRA, (int)width, (int)height, GL_BGRA, GL_UNSIGNED_BYTE, 0, &meTextureRef); GPUImageTextureInput *meTextureInput = [[GPUImageTextureInput alloc] initWithTexture:CVOpenGLESTextureGetName(meTextureRef) size:CGSizeMake(width, height)]; GPUImageBrightnessFilter *filter = [[GPUImageBrightnessFilter alloc] init]; filter.brightness = 0.5; [meTextureInput addTarget:filter]; [filter setFrameProcessingCompletionBlock:^(GPUImageOutput *imageOutput, CMTime time) { [asyncVideoCompositionRequest finishWithComposedVideoFrame:((GPUImageBrightnessFilter *)imageOutput).renderTarget]; }]; [meTextureInput processTextureWithFrameTime:kCMTimeZero]; CFRelease(meTextureRef); CVOpenGLESTextureCacheFlush(_textureCache, 0); CVPixelBufferUnlockBaseAddress(mePixelBuffer, kCVPixelBufferLock_ReadOnly); } } 

I do not use GPUMovieWriter or the video API in GPUImage because I need to control my composition in more detail. The composition may consist of several instructions with color keys that refer to different green overlays of the video in different time ranges, and it seems to me that the movie APIs in GPUImage limited to filtering the entire video file. I also need compositional abilities to control tracks and audio mixes.

I tried wrapping my head doing all this in GL with custom shaders, but I decided that I would use existing frameworks that seem to do what I'm trying to do.

Answer 1

I wrote a class called GPUImageFrameInput, modified from GPUImageMovie. CVPixelBufferRef is its input. It goes like this:

• wrap sourcePixelBuffer for GPUImageFrameOutput
• create GPUImageFrameInput wrapped destinationPixelBuffer
• complete the filter chain
• done

Here is the key code.

 // output // wrap the sourcePixelBuffer from the request // it modified from GPUImageVideoCamera @interface GPUImageFrameOutput() { } -(void)processSourcePixelBuffer:(CVPixelBufferRef)pixelBuffer withSampleTime:(CMTime)currentSampleTime; @end @implement GPUImageFrameOutput() -(void)processSourcePixelBuffer:(CVPixelBufferRef)pixelBuffer withSampleTime:(CMTime)currentSampleTime { runSynchronouslyOnVideoProcessingQueue(^{ [GPUImageContext useImageProcessingContext]; int bufferHeight = (int) CVPixelBufferGetHeight(movieFrame); int bufferWidth = (int) CVPixelBufferGetWidth(movieFrame); if (bufferHeight == 0 || bufferWidth == 0) { return; } // almost same as // [GPUImageVideoCamera processVideoSampleBuffer:] // } @end // input // wrap the destinationPixelBuffer @interface GPUImageFrameInput() { CVPixelBufferRef targetBuffer; // ... others } @end - (void)setPixelBuffer:(CVPixelBufferRef)buffer{ targetBuffer = buffer; } - (CVOpenGLESTextureRef)createDataFBO { if (!movieFramebuffer) { glActiveTexture(GL_TEXTURE1); glGenFramebuffers(1, &movieFramebuffer); glBindFramebuffer(GL_FRAMEBUFFER, movieFramebuffer); } glBindFramebuffer(GL_FRAMEBUFFER, movieFramebuffer); glViewport(0, 0, (int)_videoSize.width, (int)_videoSize.height); CVOpenGLESTextureRef renderTexture = nil; if ([GPUImageContext supportsFastTextureUpload]) { CVBufferSetAttachment(targetBuffer, kCVImageBufferColorPrimariesKey, kCVImageBufferColorPrimaries_ITU_R_709_2, kCVAttachmentMode_ShouldPropagate); CVBufferSetAttachment(targetBuffer, kCVImageBufferYCbCrMatrixKey, kCVImageBufferYCbCrMatrix_ITU_R_601_4, kCVAttachmentMode_ShouldPropagate); CVBufferSetAttachment(targetBuffer, kCVImageBufferTransferFunctionKey, kCVImageBufferTransferFunction_ITU_R_709_2, kCVAttachmentMode_ShouldPropagate); CVOpenGLESTextureCacheCreateTextureFromImage (kCFAllocatorDefault, [[GPUImageContext sharedImageProcessingContext] coreVideoTextureCache], targetBuffer, NULL, // texture attributes GL_TEXTURE_2D, GL_RGBA, // opengl format (int)CVPixelBufferGetWidth(targetBuffer), (int)CVPixelBufferGetHeight(targetBuffer), GL_BGRA, // native iOS format GL_UNSIGNED_BYTE, 0, &renderTexture); glBindTexture(CVOpenGLESTextureGetTarget(renderTexture), CVOpenGLESTextureGetName(renderTexture)); glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE); glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE); glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, CVOpenGLESTextureGetName(renderTexture), 0); } else { //... } GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER); NSAssert(status == GL_FRAMEBUFFER_COMPLETE, @"Incomplete filter FBO: %d", status); return renderTexture; } 

then you can create a gpuimage chain like the others.

 [frameInput setPixelBuffer:destinationPixelBuffer]; for (...) { GPUImageMovieFrameOutput *output ... [output addTarget:filter atTextureLocation:index]; } [filter addTarget:frameInput]; 

hope for help!