从CMBlockBuffer中提取h264

我正在使用Apple VideoTool Box（iOS）压缩设备相机捕获的原始帧。

我的callback被包含CMBlockBuffer的CMSampleBufferRef对象调用。

CMBlockBuffer对象包含H264基本stream，但是我没有find任何方法来获得指向基本stream的指针。

当我打印到控制台的CMSampleBufferRef对象时，我得到：

(lldb) po blockBufferRef CMBlockBuffer 0x1701193e0 totalDataLength: 4264 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2 [0] 4264 bytes @ offset 128 Buffer Reference: CMBlockBuffer 0x170119350 totalDataLength: 4632 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2 [0] 4632 bytes @ offset 0 Memory Block 0x10295c000, 4632 bytes (custom V=0 A=0x0 F=0x18498bb44 R=0x0) 

看来，我设法得到指针的CMBlockBuffer对象包含另一个CMBlockBuferRef（4632字节），这是不可访问的。

任何人都可以发布如何访问H264 elemantrystream？

谢谢！

Answer 1

我一直在为自己奋斗了一段时间，终于弄清楚了一切。

函数CMBlockBufferGetDataPointer让你访问所有你需要的数据，但是有一些不是很明显的事情需要做，把它转换成基本stream。

AVCC与附件B格式

CMBlockBuffer中的数据以AVCC格式存储，而基本stream通常遵循附录B规范（ 这里是两种格式的极好概述）。 在AVCC格式中，前4个字节包含NAL单元的长度（H264包的另一个字）。 您需要用4字节的起始代码replace这个头：0x00 0x00 0x00 0x01，作为附件B基本stream中的NAL单元之间的分隔符（3字节版本0x00 0x00 0x01也可以正常工作）。

单个CMBlockBuffer中的多个NAL单元

下一个不是很明显的事情是，一个单独的CMBlockBuffer有时会包含多个NAL单元。 苹果公司似乎为每个I-Frame NAL单元（也称为IDR）添加了一个包含元数据的附加NAL单元（SEI）。 这可能是你在一个CMBlockBuffer对象中看到多个缓冲区的原因。 但是， CMBlockBufferGetDataPointer函数为您提供了一个可访问所有数据的指针。 也就是说，多个NAL单元的存在使得AVCC标题的转换复杂化。 现在，您实际上必须读取包含在AVCC标头中的长度值才能find下一个NAL单元，并继续转换标头，直到达到缓冲区末尾。

Big-Endian与Little-Endian

接下来不是很明显的事情是，AVCC头以Big-Endian格式存储，而iOS本身就是Little-Endian。 所以当你读取AVCC头文件中包含的长度值时，首先将它传递给CFSwapInt32BigToHost函数。

SPS和PPS NAL单位

最后不是很明显的是CMBlockBuffer里面的数据不包含参数NAL单元SPS和PPS，其中包含解码器的configuration参数，如configuration文件，级别，分辨率，帧率。 这些数据以元数据forms存储在采样缓冲区的格式描述中，可以通过函数CMVideoFormatDescriptionGetH264ParameterSetAtIndex进行访问。 请注意，您必须在发送之前将起始代码添加到这些NAL单元。 SPS和PPS NAL单元不必随每个新帧一起发送。 解码器只需要读取一次，但定期重新发送它们是很常见的，例如在每个新的I帧NAL单元之前。

代码示例

以下是考虑到所有这些事情的代码示例。

 static void videoFrameFinishedEncoding(void *outputCallbackRefCon, void *sourceFrameRefCon, OSStatus status, VTEncodeInfoFlags infoFlags, CMSampleBufferRef sampleBuffer) { // Check if there were any errors encoding if (status != noErr) { NSLog(@"Error encoding video, err=%lld", (int64_t)status); return; } // In this example we will use a NSMutableData object to store the // elementary stream. NSMutableData *elementaryStream = [NSMutableData data]; // Find out if the sample buffer contains an I-Frame. // If so we will write the SPS and PPS NAL units to the elementary stream. BOOL isIFrame = NO; CFArrayRef attachmentsArray = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, 0); if (CFArrayGetCount(attachmentsArray)) { CFBooleanRef notSync; CFDictionaryRef dict = CFArrayGetValueAtIndex(attachmentsArray, 0); BOOL keyExists = CFDictionaryGetValueIfPresent(dict, kCMSampleAttachmentKey_NotSync, (const void **)&notSync); // An I-Frame is a sync frame isIFrame = !keyExists || !CFBooleanGetValue(notSync); } // This is the start code that we will write to // the elementary stream before every NAL unit static const size_t startCodeLength = 4; static const uint8_t startCode[] = {0x00, 0x00, 0x00, 0x01}; // Write the SPS and PPS NAL units to the elementary stream before every I-Frame if (isIFrame) { CMFormatDescriptionRef description = CMSampleBufferGetFormatDescription(sampleBuffer); // Find out how many parameter sets there are size_t numberOfParameterSets; CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description, 0, NULL, NULL, &numberOfParameterSets, NULL); // Write each parameter set to the elementary stream for (int i = 0; i < numberOfParameterSets; i++) { const uint8_t *parameterSetPointer; size_t parameterSetLength; CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description, i, &parameterSetPointer, &parameterSetLength, NULL, NULL); // Write the parameter set to the elementary stream [elementaryStream appendBytes:startCode length:startCodeLength]; [elementaryStream appendBytes:parameterSetPointer length:parameterSetLength]; } } // Get a pointer to the raw AVCC NAL unit data in the sample buffer size_t blockBufferLength; uint8_t *bufferDataPointer = NULL; CMBlockBufferGetDataPointer(CMSampleBufferGetDataBuffer(sampleBuffer), 0, NULL, &blockBufferLength, (char **)&bufferDataPointer); // Loop through all the NAL units in the block buffer // and write them to the elementary stream with // start codes instead of AVCC length headers size_t bufferOffset = 0; static const int AVCCHeaderLength = 4; while (bufferOffset < blockBufferLength - AVCCHeaderLength) { // Read the NAL unit length uint32_t NALUnitLength = 0; memcpy(&NALUnitLength, bufferDataPointer + bufferOffset, AVCCHeaderLength); // Convert the length value from Big-endian to Little-endian NALUnitLength = CFSwapInt32BigToHost(NALUnitLength); // Write start code to the elementary stream [elementaryStream appendBytes:startCode length:startCodeLength]; // Write the NAL unit without the AVCC length header to the elementary stream [elementaryStream appendBytes:bufferDataPointer + bufferOffset + AVCCHeaderLength length:NALUnitLength]; // Move to the next NAL unit in the block buffer bufferOffset += AVCCHeaderLength + NALUnitLength; } } 

Answer 2

感谢安东一个很好的答案！ 对于那些有兴趣使用这里讨论的概念的人来说，将Swift-based项目直接放在你的解决scheme的Swift-port上。

 public func didEncodeFrame(frame: CMSampleBuffer) { print ("Received encoded frame in delegate...") //----AVCC to Elem stream-----// var elementaryStream = NSMutableData() //1. check if CMBuffer had I-frame var isIFrame:Bool = false let attachmentsArray:CFArray = CMSampleBufferGetSampleAttachmentsArray(frame, false)! //check how many attachments if ( CFArrayGetCount(attachmentsArray) > 0 ) { let dict = CFArrayGetValueAtIndex(attachmentsArray, 0) let dictRef:CFDictionaryRef = unsafeBitCast(dict, CFDictionaryRef.self) //get value let value = CFDictionaryGetValue(dictRef, unsafeBitCast(kCMSampleAttachmentKey_NotSync, UnsafePointer<Void>.self)) if ( value != nil ){ print ("IFrame found...") isIFrame = true } } //2. define the start code let nStartCodeLength:size_t = 4 let nStartCode:[UInt8] = [0x00, 0x00, 0x00, 0x01] //3. write the SPS and PPS before I-frame if ( isIFrame == true ){ let description:CMFormatDescriptionRef = CMSampleBufferGetFormatDescription(frame)! //how many params var numParams:size_t = 0 CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description, 0, nil, nil, &numParams, nil) //write each param-set to elementary stream print("Write param to elementaryStream ", numParams) for i in 0..<numParams { var parameterSetPointer:UnsafePointer<UInt8> = nil var parameterSetLength:size_t = 0 CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description, i, &parameterSetPointer, &parameterSetLength, nil, nil) elementaryStream.appendBytes(nStartCode, length: nStartCodeLength) elementaryStream.appendBytes(parameterSetPointer, length: unsafeBitCast(parameterSetLength, Int.self)) } } //4. Get a pointer to the raw AVCC NAL unit data in the sample buffer var blockBufferLength:size_t = 0 var bufferDataPointer: UnsafeMutablePointer<Int8> = nil CMBlockBufferGetDataPointer(CMSampleBufferGetDataBuffer(frame)!, 0, nil, &blockBufferLength, &bufferDataPointer) print ("Block length = ", blockBufferLength) //5. Loop through all the NAL units in the block buffer var bufferOffset:size_t = 0 let AVCCHeaderLength:Int = 4 while (bufferOffset < (blockBufferLength - AVCCHeaderLength) ) { // Read the NAL unit length var NALUnitLength:UInt32 = 0 memcpy(&NALUnitLength, bufferDataPointer + bufferOffset, AVCCHeaderLength) //Big-Endian to Little-Endian NALUnitLength = CFSwapInt32(NALUnitLength) if ( NALUnitLength > 0 ){ print ( "NALUnitLen = ", NALUnitLength) // Write start code to the elementary stream elementaryStream.appendBytes(nStartCode, length: nStartCodeLength) // Write the NAL unit without the AVCC length header to the elementary stream elementaryStream.appendBytes(bufferDataPointer + bufferOffset + AVCCHeaderLength, length: Int(NALUnitLength)) // Move to the next NAL unit in the block buffer bufferOffset += AVCCHeaderLength + size_t(NALUnitLength); print("Moving to next NALU...") } } print("Read completed...") }