me 学习 MacOS 开发 (Part 24: CVPixelBuffer)

学习 MacOS 开发 (Part 24: CVPixelBuffer)

Open nonocast opened this issue 2 years ago • 0 comments

Overview

Core Video

CVBuffer
CVImageBuffer
CVPixelBuffer (typealias CVPixelBuffer = CVImageBuffer)
CVPixelBufferPool
CVPixelFormatDescription
CVTime

Core Media

CMSampleBuffer (在压缩后，CMSampleBuffer中包含CMBlockBuffer，当表达压缩前则包含CVPixelBuffer)
CMBlockBuffer (在抽象顶层，表示任意buffer，多指压缩后数据)
CMFormatDescription
CMAttachment
CMTime
CMTimeRange
CMTimeMapping
CMBlockCM

VideoToolbox

VTCompressionSession
VTCompressionSessionCreate(allocator:width:height:codecType:encoderSpecification:imageBufferAttributes:compressedDataAllocator:outputCallback:refcon:compressionSessionOut:)
VTCompressionSessionEncodeFrame(_:imageBuffer:presentationTimeStamp:duration:frameProperties:sourceFrameRefcon:infoFlagsOut:)
compressionOutputCallback(outputCallbackRefCon _: UnsafeMutableRawPointer?, sourceFrameRefCon _: UnsafeMutableRawPointer?, status _: OSStatus, infoFlags _: VTEncodeInfoFlags, sampleBuffer _: CMSampleBuffer?) -> Swift.Void
VTCreateCGImageFromCVPixelBuffer(_:options:imageOut:)

从上述关系可以发现:

如果表示一张图片pixel buffer采用CVPixelBuffer
如果表示一个压缩数据则采用CMSampleBuffer
VideoToolbox encode时进CVPixelBuffer出CMSampleBuffer
AVFoundation在Capture时出的是CMSampleBuffer

我们看下如何磁盘上一张JPG和CVPixelBuffer的关系。

Creating Pixel Buffers

CVPixelBufferCreate(::::::) - Creates a single pixel buffer for a given size and pixel format.
CVPixelBufferCreateWithBytes(::::::::::) - Creates a pixel buffer for a given size and pixel format containing data specified by a memory location.
CVPixelBufferCreateWithPlanarBytes(::::::::::::::_:) - Creates a single pixel buffer in planar format for a given size and pixel format containing data specified by a memory location.
CVPixelBufferCreateWithIOSurface(::::) - Call to create a single pixel buffer for a passed-in IOSurface.

Planar and Packed

一个图像可以理解成pixel的二维数组
每一个pixel都可以用不同的color space进行描述，如RGB, CMY, CMYK, YUV, HSV, HSI, LAB
然后还需要确定字节保存方式，假设一个pixel对应a, b, c三个参数，如果是RGB，则a为R, b为G, c为B, 如果是YUV，则a为Y, b为U, c为V
如果字节排列是一个像素一个像素的方式，即(a00, b00, c00, a01, b01, c01....)，称为packed format
如果字节是根据参数排列，即(a00, a01, a02...b00, b01, b02...c00, c01, c02)，成为planar，如果是(a00, a01, a02...b00, c00, b01, c01, b02, c02...)，这种称为semi-planar
packed容易理解，planar对程序更加友好，可以根据planar动态裁剪，比如读1/3就可以得到图片的大概样子，容易增加planar来细化图像
crazybiocomputing: RGB file: Packed or Planar?
colors - What is difference between planar, semi planar and interleaved format.? - Stack Overflow

CVPixelBuffer

CVPixelBuffer | Apple Developer Documentation

An image buffer that holds pixels in main memory. A Core Video pixel buffer is an image buffer that holds pixels in main memory. Applications generating frames, compressing or decompressing video, or using Core Image can all make use of Core Video pixel buffers.

简而言之，CVPixelBuffer其实就是图像的buffer，图像有很多种像素表达方式

/*
CoreVideo pixel format type constants.
CoreVideo does not provide support for all of these formats; this list just defines their names.
*/
#if COREVIDEO_USE_DERIVED_ENUMS_FOR_CONSTANTS
enum : OSType
#else
enum
#endif
{
  kCVPixelFormatType_1Monochrome    = 0x00000001, /* 1 bit indexed */
  kCVPixelFormatType_2Indexed       = 0x00000002, /* 2 bit indexed */
  kCVPixelFormatType_4Indexed       = 0x00000004, /* 4 bit indexed */
  kCVPixelFormatType_8Indexed       = 0x00000008, /* 8 bit indexed */
  kCVPixelFormatType_1IndexedGray_WhiteIsZero = 0x00000021, /* 1 bit indexed gray, white is zero */
  kCVPixelFormatType_2IndexedGray_WhiteIsZero = 0x00000022, /* 2 bit indexed gray, white is zero */
  kCVPixelFormatType_4IndexedGray_WhiteIsZero = 0x00000024, /* 4 bit indexed gray, white is zero */
  kCVPixelFormatType_8IndexedGray_WhiteIsZero = 0x00000028, /* 8 bit indexed gray, white is zero */
  kCVPixelFormatType_16BE555        = 0x00000010, /* 16 bit BE RGB 555 */
  kCVPixelFormatType_16LE555        = 'L555',     /* 16 bit LE RGB 555 */
  kCVPixelFormatType_16LE5551       = '5551',     /* 16 bit LE RGB 5551 */
  kCVPixelFormatType_16BE565        = 'B565',     /* 16 bit BE RGB 565 */
  kCVPixelFormatType_16LE565        = 'L565',     /* 16 bit LE RGB 565 */
  kCVPixelFormatType_24RGB          = 0x00000018, /* 24 bit RGB */
  kCVPixelFormatType_24BGR          = '24BG',     /* 24 bit BGR */
  kCVPixelFormatType_32ARGB         = 0x00000020, /* 32 bit ARGB */
  kCVPixelFormatType_32BGRA         = 'BGRA',     /* 32 bit BGRA */
  kCVPixelFormatType_32ABGR         = 'ABGR',     /* 32 bit ABGR */
  kCVPixelFormatType_32RGBA         = 'RGBA',     /* 32 bit RGBA */
  kCVPixelFormatType_64ARGB         = 'b64a',     /* 64 bit ARGB, 16-bit big-endian samples */
  kCVPixelFormatType_48RGB          = 'b48r',     /* 48 bit RGB, 16-bit big-endian samples */
  kCVPixelFormatType_32AlphaGray    = 'b32a',     /* 32 bit AlphaGray, 16-bit big-endian samples, black is zero */
  kCVPixelFormatType_16Gray         = 'b16g',     /* 16 bit Grayscale, 16-bit big-endian samples, black is zero */
  kCVPixelFormatType_30RGB          = 'R10k',     /* 30 bit RGB, 10-bit big-endian samples, 2 unused padding bits (at least significant end). */
  kCVPixelFormatType_422YpCbCr8     = '2vuy',     /* Component Y'CbCr 8-bit 4:2:2, ordered Cb Y'0 Cr Y'1 */
  kCVPixelFormatType_4444YpCbCrA8   = 'v408',     /* Component Y'CbCrA 8-bit 4:4:4:4, ordered Cb Y' Cr A */
  kCVPixelFormatType_4444YpCbCrA8R  = 'r408',     /* Component Y'CbCrA 8-bit 4:4:4:4, rendering format. full range alpha, zero biased YUV, ordered A Y' Cb Cr */
  kCVPixelFormatType_4444AYpCbCr8   = 'y408',     /* Component Y'CbCrA 8-bit 4:4:4:4, ordered A Y' Cb Cr, full range alpha, video range Y'CbCr. */
  kCVPixelFormatType_4444AYpCbCr16  = 'y416',     /* Component Y'CbCrA 16-bit 4:4:4:4, ordered A Y' Cb Cr, full range alpha, video range Y'CbCr, 16-bit little-endian samples. */
  kCVPixelFormatType_444YpCbCr8     = 'v308',     /* Component Y'CbCr 8-bit 4:4:4 */
  kCVPixelFormatType_422YpCbCr16    = 'v216',     /* Component Y'CbCr 10,12,14,16-bit 4:2:2 */
  kCVPixelFormatType_422YpCbCr10    = 'v210',     /* Component Y'CbCr 10-bit 4:2:2 */
  kCVPixelFormatType_444YpCbCr10    = 'v410',     /* Component Y'CbCr 10-bit 4:4:4 */
  kCVPixelFormatType_420YpCbCr8Planar = 'y420',   /* Planar Component Y'CbCr 8-bit 4:2:0.  baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrPlanar struct */
  kCVPixelFormatType_420YpCbCr8PlanarFullRange    = 'f420',   /* Planar Component Y'CbCr 8-bit 4:2:0, full range.  baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrPlanar struct */
  kCVPixelFormatType_422YpCbCr_4A_8BiPlanar = 'a2vy', /* First plane: Video-range Component Y'CbCr 8-bit 4:2:2, ordered Cb Y'0 Cr Y'1; second plane: alpha 8-bit 0-255 */
  kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange = '420v', /* Bi-Planar Component Y'CbCr 8-bit 4:2:0, video-range (luma=[16,235] chroma=[16,240]).  baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct */
  kCVPixelFormatType_420YpCbCr8BiPlanarFullRange  = '420f', /* Bi-Planar Component Y'CbCr 8-bit 4:2:0, full-range (luma=[0,255] chroma=[1,255]).  baseAddr points to a big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct */ 
  kCVPixelFormatType_422YpCbCr8_yuvs = 'yuvs',     /* Component Y'CbCr 8-bit 4:2:2, ordered Y'0 Cb Y'1 Cr */
  kCVPixelFormatType_422YpCbCr8FullRange = 'yuvf', /* Component Y'CbCr 8-bit 4:2:2, full range, ordered Y'0 Cb Y'1 Cr */
  kCVPixelFormatType_OneComponent8  = 'L008',     /* 8 bit one component, black is zero */
  kCVPixelFormatType_TwoComponent8  = '2C08',     /* 8 bit two component, black is zero */
  kCVPixelFormatType_30RGBLEPackedWideGamut = 'w30r', /* little-endian RGB101010, 2 MSB are zero, wide-gamut (384-895) */
  kCVPixelFormatType_OneComponent16Half  = 'L00h',     /* 16 bit one component IEEE half-precision float, 16-bit little-endian samples */
  kCVPixelFormatType_OneComponent32Float = 'L00f',     /* 32 bit one component IEEE float, 32-bit little-endian samples */
  kCVPixelFormatType_TwoComponent16Half  = '2C0h',     /* 16 bit two component IEEE half-precision float, 16-bit little-endian samples */
  kCVPixelFormatType_TwoComponent32Float = '2C0f',     /* 32 bit two component IEEE float, 32-bit little-endian samples */
  kCVPixelFormatType_64RGBAHalf          = 'RGhA',     /* 64 bit RGBA IEEE half-precision float, 16-bit little-endian samples */
  kCVPixelFormatType_128RGBAFloat        = 'RGfA',     /* 128 bit RGBA IEEE float, 32-bit little-endian samples */
  kCVPixelFormatType_14Bayer_GRBG        = 'grb4',     /* Bayer 14-bit Little-Endian, packed in 16-bits, ordered G R G R... alternating with B G B G... */
  kCVPixelFormatType_14Bayer_RGGB        = 'rgg4',     /* Bayer 14-bit Little-Endian, packed in 16-bits, ordered R G R G... alternating with G B G B... */
  kCVPixelFormatType_14Bayer_BGGR        = 'bgg4',     /* Bayer 14-bit Little-Endian, packed in 16-bits, ordered B G B G... alternating with G R G R... */
  kCVPixelFormatType_14Bayer_GBRG        = 'gbr4',     /* Bayer 14-bit Little-Endian, packed in 16-bits, ordered G B G B... alternating with R G R G... */
};

一定要注意图像格式的兼容性问题，比如当不指定时pixel format是kCVPixelFormatType_422YpCbCr8，通过如下代码可改为kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange

    let videoOutput = AVCaptureVideoDataOutput()
    videoOutput.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange]

当图像在不同层面，比如capture和encoder之间，如何理解pixel layout其实就是一个关键因素，一旦没有协调好，大家就无法理解buffer内容，一定要注意。

常用的pixel format:

RGB
- kCVPixelFormatType_32BGRA = 'BGRA',
- kCVPixelFormatType_32BGRA = 'BGRA',
- kCVPixelFormatType_32ABGR = 'ABGR',
- kCVPixelFormatType_32RGBA = 'RGBA',
NV12
- kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange = '420v'
- kCVPixelFormatType_420YpCbCr8BiPlanarFullRange = '420f'
YUV420P
- kCVPixelFormatType_420YpCbCr8Planar = 'y420'

使用时:

如果需要通过指针直接操作buffer，则需要显示lock和unlock
如果是转换成CIImage, CGImage，则不需要lock

所谓lock, unlock其实就是pthread的lock,避免多个线程同时操作内存。

参考阅读

May 13 '22 01:05 nonocast

me me copied to clipboard

学习 MacOS 开发 (Part 24: CVPixelBuffer)

Overview

Core Video

Core Media

VideoToolbox

CVPixelBuffer

参考阅读

me
me copied to clipboard