学习 MacOS 开发 (Part 17: Image)

[nonocast]

Overview

• CMSampleBufferRef (Core Media)
• CVPixelBufferRef (Core Video)
• CIImage (Core Image)
• CGImage (Core Graphics)
• UIImage/NSImage (Cocoa)
• Image (SwiftUI)

Framework

来源: iOS图形处理概论：OpenGL ES，Metal，Core Graphics，Core Image，GPUImage，OpenCV等 - 掘金

• 界面图形框架: UIKit/AppKit (Cocoa)
• 核心动画框架: Core Animation
• 苹果封装的图形框架: Core Graphics & Quartz 2D
• 传统跨平台图形框架: OpenGL ES
• 苹果最新力推的图形框架: Metal
• 适合图片的苹果滤镜框架: Core Image
• 适合视频的第三方滤镜方案: GPUImage
• 游戏引擎: Scene Kit (3D) 和 Sprite Kit (2D)
• 计算机视觉在iOS的应用: OpenCV for iOS

NSImage

每个kit/Framework都需要自己的Image，显然NSImage服务于Cocoa AppKit，是一个high level的图像对象，通过转换来链接外部或下游的CGImage或CIImage。

Cocoa Drawing Guide - Image

import Cocoa

class ViewController: NSViewController {
    @IBOutlet weak var imageView: NSImageView!
    
    override func viewDidLoad() {
        super.viewDidLoad()

        let image = NSImage.init(named: "Sample")
        imageView.image = image
    }
}

SwiftUI

ContentView.swift

struct ContentView: View {
  @StateObject var model = ContentViewModel()
  
  var body: some View {
    if let image = model.image {
      Image(image, scale: 1.0, orientation: .up, label: Text("image"))
        .resizable().scaledToFit().padding()
        .foregroundColor(.init(NSColor(calibratedRed: 0.9, green: 0.8, blue: 0.8, alpha: 1.0)))
      
    } else {
      EmptyView()
    }
  }
}

class ContentViewModel : NSObject, ObservableObject {
  @Published var image:CGImage?
  
  override init() {
    super.init()
//    let nsImage = NSImage(named: "dance")
// disable sandbox or grant permission
    let nsImage = NSImage(byReferencingFile: "/Users/nonocast/Pictures/Samples/dance.jpg")
    if let p = nsImage {
      image = p.cgImage(forProposedRect: nil, context: nil, hints: nil)
    }
  }
}

struct ContentView_Previews: PreviewProvider {
  static var previews: some View {
    ContentView().frame(width:500, height:500)
  }
}

Core Graphics

Core Graphics等同于Quartz 2D，整个MacOS系统绘图分为Quartz和OpenGL两个路线，Cocoa (AppKIt and UIKit)都是based on Quartz, 所以两个层面的习惯非常一致。基本上NSImage和UIImage就是对CGImage的封装。OpenGL部分则封装在GLKit。

Note: If a UIKit method or function exists to accomplish a given image-related task, you should use it instead of any corresponding lower-level function.

Loading Images

所以，应该优先使用UIImage和NSImage，在需要low-level task的时候再去考虑CGImage， you can access the CGImageRef object backing a UIImage object through the CGImage property。

The Core Graphics framework of Quartz is the most important of the lower-level system frameworks. Several of its functions correspond to UIKit functions and methods; for example, some Core Graphics functions allow you to create and draw to bitmap graphics contexts, while others let you create images from various sources. However, Core Graphics offers more options for handling images. With Core Graphics you can create and apply image masks, create images from portions of existing images, apply color spaces, and access a number of additional image attributes, including bytes per row, bits per pixel, and rendering intent.

The Image I/O framework is closely related to Core Graphics. It allows an app to read and write most image file formats, including the standard web formats, high dynamic range images, and raw camera data. It features fast image encoding and decoding, image metadata, and image caching.

Assets Library is a framework that allows an app to access assets managed by the Photos app. You can get an asset either by representation (for example, PNG or JPEG) or URL. From the representation or URL you can obtain a Core Graphics image object or the raw image data. The framework also lets you write images to the Saved Photos Album.

Core Image

Apple - About Core Image

用来做运行时图像处理，比如实时滤镜等效果，Core Image based on Metal。只有需要图像efffect时就可以拿出来用。

Data and NSData

The Data value type allows simple byte buffers to take on the behavior of Foundation objects. You can create empty or pre-populated buffers from a variety of sources and later add or remove bytes. You can filter and sort the content, or compare against other buffers. You can manipulate subranges of bytes and iterate over some or all of them.

Data bridges to the NSData class and its mutable subclass, NSMutableData. You can use these interchangeably in code that interacts with Objective-C APIs.

• Data: A byte buffer in memory., 是定义在Foundation中，所以使用时需要import Foundation
• NSData属于Cocoa范畴

Data and NSData. Two types of Data? | by Steven Curtis | Medium

import Foundation

let str = "abc"
let data = Data(str.utf8)
// let data: Data = .init(str.utf8)

print(data) // 3 bytes
print(data as NSData) // {length = 3, bytes = 0x616263 }
print(data.map { String(format: "%02x", $0) }.joined()) // 616263
print(data.map { String(format: "0x%02x", $0) }.joined(separator: " ")) // 0x61 0x62 0x63

参考文档

• Loading Images
• Quartz 2D Programming Guide
• iOS绘图 - 吵了个吵
• XCode编程读取图片、视频、摄像头 - 知乎
• 从RGB数组到CVPixelBuffer和UIImage再到CoreML - 知乎
• iOS - 图形高级处理 (一、图片显示相关理论) - 掘金
• iOS-图片高级处理（三、图片处理实践） - 掘金