如何直接旋转iOS 4中的CVImageBuffer图像而不转换为UIImage？

我在iPhone上使用OpenCV 2.2来检测脸部。 我使用IOS 4的AVCaptureSession来访问相机stream，如下面的代码所示。

我的挑战是video帧以CVBufferRef（指向CVImageBuffer的指针）对象的forms出现，它们以480px宽，300px高的风景来定位。 如果你侧身拿着手机，这很好，但是当手机保持直立状态时，我想顺时针旋转这些框架90度，这样OpenCV就可以正确地find脸部。

我可以将CVBufferRef转换为CGImage，然后转换为UIImage，然后像这个人一样旋转 ： 旋转从video帧中获取的CGImage

但是，这浪费了很多的CPU。 我正在寻找更快的方式来旋转图像进来，理想情况下使用GPU来做这个处理，如果可能的话。

有任何想法吗？

伊恩

代码示例： 

-(void) startCameraCapture { // Start up the face detector faceDetector = [[FaceDetector alloc] initWithCascade:@"haarcascade_frontalface_alt2" withFileExtension:@"xml"]; // Create the AVCapture Session session = [[AVCaptureSession alloc] init]; // create a preview layer to show the output from the camera AVCaptureVideoPreviewLayer *previewLayer = [AVCaptureVideoPreviewLayer layerWithSession:session]; previewLayer.frame = previewView.frame; previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill; [previewView.layer addSublayer:previewLayer]; // Get the default camera device AVCaptureDevice* camera = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo]; // Create a AVCaptureInput with the camera device NSError *error=nil; AVCaptureInput* cameraInput = [[AVCaptureDeviceInput alloc] initWithDevice:camera error:&error]; if (cameraInput == nil) { NSLog(@"Error to create camera capture:%@",error); } // Set the output AVCaptureVideoDataOutput* videoOutput = [[AVCaptureVideoDataOutput alloc] init]; videoOutput.alwaysDiscardsLateVideoFrames = YES; // create a queue besides the main thread queue to run the capture on dispatch_queue_t captureQueue = dispatch_queue_create("catpureQueue", NULL); // setup our delegate [videoOutput setSampleBufferDelegate:self queue:captureQueue]; // release the queue. I still don't entirely understand why we're releasing it here, // but the code examples I've found indicate this is the right thing. Hmm... dispatch_release(captureQueue); // configure the pixel format videoOutput.videoSettings = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA], (id)kCVPixelBufferPixelFormatTypeKey, nil]; // and the size of the frames we want // try AVCaptureSessionPresetLow if this is too slow... [session setSessionPreset:AVCaptureSessionPresetMedium]; // If you wish to cap the frame rate to a known value, such as 10 fps, set // minFrameDuration. videoOutput.minFrameDuration = CMTimeMake(1, 10); // Add the input and output [session addInput:cameraInput]; [session addOutput:videoOutput]; // Start the session [session startRunning]; } - (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection { // only run if we're not already processing an image if (!faceDetector.imageNeedsProcessing) { // Get CVImage from sample buffer CVImageBufferRef cvImage = CMSampleBufferGetImageBuffer(sampleBuffer); // Send the CVImage to the FaceDetector for later processing [faceDetector setImageFromCVPixelBufferRef:cvImage]; // Trigger the image processing on the main thread [self performSelectorOnMainThread:@selector(processImage) withObject:nil waitUntilDone:NO]; } }

vImage是一个相当快的方法来做到这一点。 不过需要ios5。 这个调用说ARGB，但它适用于你从缓冲区获得的BGRA。

这也有一个好处，你可以切出一部分缓冲区并旋转。 在这里看到我的答案 

 - (unsigned char*) rotateBuffer: (CMSampleBufferRef) sampleBuffer { CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer); CVPixelBufferLockBaseAddress(imageBuffer,0); size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer); size_t width = CVPixelBufferGetWidth(imageBuffer); size_t height = CVPixelBufferGetHeight(imageBuffer); size_t currSize = bytesPerRow*height*sizeof(unsigned char); size_t bytesPerRowOut = 4*height*sizeof(unsigned char); void *srcBuff = CVPixelBufferGetBaseAddress(imageBuffer); unsigned char *outBuff = (unsigned char*)malloc(currSize); vImage_Buffer ibuff = { srcBuff, height, width, bytesPerRow}; vImage_Buffer ubuff = { outBuff, width, height, bytesPerRowOut}; uint8_t rotConst = 1; // 0, 1, 2, 3 is equal to 0, 90, 180, 270 degrees rotation vImage_Error err= vImageRotate90_ARGB8888 (&ibuff, &ubuff, NULL, rotConst, NULL,0); if (err != kvImageNoError) NSLog(@"%ld", err); return outBuff; }

也许更容易设置您想要的video方向： 

 connection.videoOrientation = AVCaptureVideoOrientationPortrait

这样你就不需要做那个旋转的噱头了

如果你旋转90度停止，那么你可以在内存中做。 以下是仅仅将数据复制到新像素缓冲区的示例代码。 做一个powershell旋转应该是直截了当的。 

 - (CVPixelBufferRef) rotateBuffer: (CMSampleBufferRef) sampleBuffer { CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer); CVPixelBufferLockBaseAddress(imageBuffer,0); size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer); size_t width = CVPixelBufferGetWidth(imageBuffer); size_t height = CVPixelBufferGetHeight(imageBuffer); void *src_buff = CVPixelBufferGetBaseAddress(imageBuffer); NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey, [NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey, nil]; CVPixelBufferRef pxbuffer = NULL; //CVReturn status = CVPixelBufferPoolCreatePixelBuffer (NULL, _pixelWriter.pixelBufferPool, &pxbuffer); CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, width, height, kCVPixelFormatType_32BGRA, (CFDictionaryRef) options, &pxbuffer); NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL); CVPixelBufferLockBaseAddress(pxbuffer, 0); void *dest_buff = CVPixelBufferGetBaseAddress(pxbuffer); NSParameterAssert(dest_buff != NULL); int *src = (int*) src_buff ; int *dest= (int*) dest_buff ; size_t count = (bytesPerRow * height) / 4 ; while (count--) { *dest++ = *src++; } //Test straight copy. //memcpy(pxdata, baseAddress, width * height * 4) ; CVPixelBufferUnlockBaseAddress(pxbuffer, 0); CVPixelBufferUnlockBaseAddress(imageBuffer, 0); return pxbuffer; }

然后，您可以使用AVAssetWriterInputPixelBufferAdaptor，如果您将其写回到AVAssetWriterInput。

以上没有优化。 您可能想要寻找更有效的复制algorithm。 一个好的开始就是就地matrix转置 。 你也想使用像素缓冲池，而不是每次创build一个新的。

编辑。 你可以使用GPU来做到这一点。 这听起来像是大量的数据被推送。 在CVPixelBufferRef中有关键的kCVPixelBufferOpenGLCompatibilityKey。 我假设你可以从CVImageBufferRef（这只是一个像素缓冲区参考）创build一个OpenGL兼容的图像，并通过着色器。 再次，海盗IMO。 您可能会看到BLAS或LAPACK是否存在“不在场”的转置方法。 如果他们这样做，那么你可以放心，他们是高度优化的。

90 CW其中new_width = width …这将为您带来一幅面向肖像的图像。 

 for (int i = 1; i <= new_height; i++) { for (int j = new_width - 1; j > -1; j--) { *dest++ = *(src + (j * width) + i) ; } }

我知道这是一个很老的问题，但最近我一直在解决类似的问题，也许有人可以find我的解决scheme有用。

我需要从iPhone摄像头（从[AVCaptureVideoDataOutput.availableVideoCVPixelFormatTypes firstObject]获取）从YCbCr格式的图像缓冲区提取原始图像数据，丢弃信息，如标题，元信息等传递给进一步处理。

另外，我需要在捕获的video帧的中心只提取一小块区域，所以需要一些裁剪。

我的条件允许只以横向拍摄video，但是当设备以横向左侧定位时，图像会上下颠倒，因此我需要在两个轴上翻转。 在图像翻转的情况下，我的想法是以相反的顺序从源图像缓冲区复制数据，并在读取数据的每一行中反转字节以在两个轴上翻转图像。 这个想法真的有效，而且我需要从源缓冲区复制数据，如果从开始或结束读取，似乎没有太多的性能损失（当然，更大的图像=更长的处理，但我处理的是非常小的数字） 。

我想知道别人怎么看这个解决scheme，当然还有一些提示如何改进代码： 

 /// Lock pixel buffer CVPixelBufferLockBaseAddress(imageBuffer, 0); /// Address where image buffer starts uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer); /// Read image parameters size_t width = CVPixelBufferGetWidth(imageBuffer); size_t height = CVPixelBufferGetHeight(imageBuffer); /// See whether image is flipped upside down BOOL isFlipped = (_previewLayer.connection.videoOrientation == AVCaptureVideoOrientationLandscapeLeft); /// Calculate cropping frame. Crop to scanAreaSize (defined as CGSize constant elsewhere) from the center of an image CGRect cropFrame = CGRectZero; cropFrame.size = scanAreaSize; cropFrame.origin.x = (width / 2.0f) - (scanAreaSize.width / 2.0f); cropFrame.origin.y = (height / 2.0f) - (scanAreaSize.height / 2.0f); /// Update proportions to cropped size width = (size_t)cropFrame.size.width; height = (size_t)cropFrame.size.height; /// Allocate memory for output image data. W*H for Y component, W*H/2 for CbCr component size_t bytes = width * height + (width * height / 2); uint8_t *outputDataBaseAddress = (uint8_t *)malloc(bytes); if(outputDataBaseAddress == NULL) { /// Memory allocation failed, unlock buffer and give up CVPixelBufferUnlockBaseAddress(imageBuffer, 0); return NULL; } /// Get parameters of YCbCr pixel format CVPlanarPixelBufferInfo_YCbCrBiPlanar *bufferInfo = (CVPlanarPixelBufferInfo_YCbCrBiPlanar *)baseAddress; NSUInteger bytesPerRowY = EndianU32_BtoN(bufferInfo->componentInfoY.rowBytes); NSUInteger offsetY = EndianU32_BtoN(bufferInfo->componentInfoY.offset); NSUInteger bytesPerRowCbCr = EndianU32_BtoN(bufferInfo->componentInfoCbCr.rowBytes); NSUInteger offsetCbCr = EndianU32_BtoN(bufferInfo->componentInfoCbCr.offset); /// Copy image data only, skipping headers and metadata. Create single buffer which will contain Y component data /// followed by CbCr component data. /// Process Y component /// Pointer to the source buffer uint8_t *src; /// Pointer to the destination buffer uint8_t *destAddress; /// Calculate crop rect offset. Crop offset is number of rows (y * bytesPerRow) + x offset. /// If image is flipped, then read buffer from the end to flip image vertically. End address is height-1! int flipOffset = (isFlipped) ? (int)((height - 1) * bytesPerRowY) : 0; int cropOffset = (int)((cropFrame.origin.y * bytesPerRowY) + flipOffset + cropFrame.origin.x); /// Set source pointer to Y component buffer start address plus crop rect offset src = baseAddress + offsetY + cropOffset; for(int y = 0; y < height; y++) { /// Copy one row of pixel data from source into the output buffer. destAddress = (outputDataBaseAddress + y * width); memcpy(destAddress, src, width); if(isFlipped) { /// Reverse bytes in row to flip image horizontally [self reverseBytes:destAddress bytesSize:(int)width]; /// Move one row up src -= bytesPerRowY; } else { /// Move to the next row src += bytesPerRowY; } } /// Calculate crop offset for CbCr component flipOffset = (isFlipped) ? (int)(((height - 1) / 2) * bytesPerRowCbCr) : 0; cropOffset = (int)((cropFrame.origin.y * bytesPerRowCbCr) + flipOffset + cropFrame.origin.x); /// Set source pointer to the CbCr component offset + crop offset src = (baseAddress + offsetCbCr + cropOffset); for(int y = 0; y < (height / 2); y++) { /// Copy one row of pixel data from source into the output buffer. destAddress = (outputDataBaseAddress + (width * height) + y * width); memcpy(destAddress, src, width); if(isFlipped) { /// Reverse bytes in row to flip image horizontally [self reverseBytes:destAddress bytesSize:(int)width]; /// Move one row up src -= bytesPerRowCbCr; } else { src += bytesPerRowCbCr; } } /// Unlock pixel buffer CVPixelBufferUnlockBaseAddress(imageBuffer, 0); /// Continue with image data in outputDataBaseAddress;
Interesting Posts