[WIP] Direct surface publishing using SyphonServerBase

[vade]

This is a WIP / Proof of concept PR

This implements direct publishing of an IOSurface through SyphonServerBase class,

- (void)publishSurface:(IOSurfaceRef)surface;

You must meet the following requirements EXTERNAL to this new api

• Your surface must be global
• it must be in a supported format that syphon clients expect.

This does not currently enforce any of those safety guards, but it could be added.

Internally this abuses publish, and every time the publishSurface: method is called we re-broadcast the new incoming IOSurfaces ID to other clients.

This is clearly suboptimal, but .. its useful in some situations:

1. I have an IOSurface pool that I dont directly control, but can mark properties of (ie global)
2. I dont have, need, or want Metal or OpenGL contexts at all, as im just a stupid video app working with CVPixelBuffers directly.
3. Due to the above, I dont have want to manage GPU devices, command buffers and other shite, I already have a surface read to go.

This also addresses issue #67

[vade]

It might make sense to make a separate server that consumes CVPixelBuffers or IOSurfaces directly, rather than the base server I admit. I was sketching in code and wanted to make my changes be as minimal as possible.

Curious your thoughts @bangnoise <3

[bangnoise]

I'm not completely against this... but...

Instantiating an IOSurface is not free, and instantiating Metal textures from them is not cheap, so as-is this is going to have a severe negative impact on clients - it might even be the case that a server blitting to Syphon's single IOSurface is faster for clients as they are now (I haven't tested). If the only motivation for this is convenience then the cost is too high.

It would need client support to cache surfaces, and for subclasses to cache their resources (eg Metal textures), and a way to retire them, probably with a (configurable) timeout as well as an explicit retire method in the server. Everything except the explicit retire could be done without changing any IPC, just the way clients behave. However existing clients would be penalised if we built this to work with them, so we might want to carefully consider that impact.

Purely on the PR as it is now, SyphonServerBase would need a way to tell subclasses the surface has been invalidated in -publishSurface:, like SyphonClientBase does with -invalidateFrame.

[vade]

So, what im doing here, is leveraging IOSurface backed video playout from AVFoundation by requesting global surfaces in an AVPlayerItemVideoOutput like so:

    let playerItemVideoOuput = AVPlayerItemVideoOutput(outputSettings: [ String(kCVPixelBufferPixelFormatTypeKey) : kCVPixelFormatType_32BGRA,
                                                                         String(kCVPixelBufferMetalCompatibilityKey) : true,
                                                                         String(kCVPixelBufferIOSurfacePropertiesKey) : [String(kIOSurfaceIsGlobal): true]] )

And then in a CVDisplayLink callback, just

    func displayLinkCallback(displayLink: CVDisplayLink) {
        
        var time = CVTimeStamp()

        CVDisplayLinkGetCurrentTime(displayLink, &time)

        let cmtime = self.playerItemVideoOuput.itemTime(for: time)
        if self.playerItemVideoOuput.hasNewPixelBuffer(forItemTime: cmtime)
        {
            var actualOutputTime = CMTime.invalid
            guard let pixelBuffer = self.playerItemVideoOuput.copyPixelBuffer(forItemTime: cmtime, itemTimeForDisplay: &actualOutputTime) else {
                return
            }
            
            // Convert CVPixelBuffer to IOSurface
            guard let surface = CVPixelBufferGetIOSurface(pixelBuffer) else { return }
            
            surface.retain()
            self.syphonServer?.publishSurface( surface.takeUnretainedValue() )
            surface.release()
        }        
    }

The client is caching in the sense that the internal pool is being re-used and surfaces are being recycled, but yes, each frame does require a publish.

From my limited testing this seems fairly fast / lightweight so far and I see performance at a surface level on par with existing client / servers.

One nuance I forgot about which this PR does not address which doing an internal blit would - flipping.

Today theres no flag to tell a client the image is flipped. And since we consume a surface 'as is' there is no opportunity to correct for flipping.

So clearly this needs some more thought :)

[vade]

I think in theory, this might be better served with a dedicated CVPixelBufferServer / CVImageBuffer server?

The API is more useful to non expert users, and in theory could sit on top of the Metal Server infrastructure, handle flipping, etc?

Thinking aloud here...

[benoitlahoz]

Hello,

I don't know if this is the place to post my question, but... I was wondering if this feature will be merged.

I'm writing a (very experimental) wrapper of Syphon for node.js, targeting Electron but not only (e.g. a Media Server that consumes its own Syphon frames to send them through WebRTC, etc.).

Electron has, since its v33, an option to get the handle from a window rendered offscreen that is actually an IOSurface. For the time being I manage to send a texture to the server by using CGLTexImageIOSurface2D in OpenGL (Metal implementation is on the roadmap) but it would be very valuable to be allowed to directly publish the surface (if it's even possible, I'm really not a specialist).

You can find the package here: node-syphon And the specific method here: CGLTexImageIOSurface2D

To try, there is an example that can be downloaded here. Switching between implementations can be made through... the 'Implementations' menu -> 'OpenGL Offscreen'.

[bangnoise]

@benoitlahoz this won't be merged as-is. The issues in my comment above would need to be addressed, plus a serious consideration of performance implications vs what we have now. We'd welcome thoughts or work.

Whether this would help you publish an Electron window or not would depend on whether the IOSurface you get is global (see kIOSurfaceIsGlobal).

[vade]

Yea, for what its worth, I internally abandoned this approach for the a metal server + BLIT internally. its faster, doesnt do dumb shit™ and enforces constraints.

I echo Toms concerns and while I love the idea of a Node / Syphon interoperability we should find a path thats maybe a touch wiser than my approach here. I may actually close this PR to avoid confusion, but I invite further discussion on the core idea.

[benoitlahoz]

Thank you @bangnoise and @vade. I would be glad to have further discussion about this. I'm working actively on this project, also digging in WebGPU for the client.

[vade]

I had a branch of Chrome web browser a while back (like 8 years ago haha) where I added Syphon support for media input. I doubt it would be helpful here, but maybe theres some thoughts as my understanding is the underlying engine is shared?

Do you require IOSurface in? Is it not possible to just make a metal texture in this branch / fork as it would be macOS anyway?

[benoitlahoz]

8 years! I remember I downloaded it (can't remember if it was the code or the app), but I unfortunately can't find it back...

For the node.js case: it doesn't look too difficult to make node and Syphon communicate, thanks to node-addon-api C++ / Objective-C++ bindings. There are a lot of packages using it (or NAPI, its C ancestor) for binding ffmpeg, gstreamer, etc. It doesn't require any IOSurface. But converting things from one to another may be costly. I actually don't know much about WebGPU, but I feel there is something to explore there to avoid this cost... Having a SyphonWebGPUServer and a SyphonWebGPUClient, even if that involves a dependency...

The big problem is on the Electron side: main (node) and renderer (chromium) processes are strictly separated for security reasons. So we have to find a way to pass texture or data (it works but ouch...) between them (main process can hold Syphon, renderer one offers user interaction, more simple to setup WebRTC, threejs, eventual mix of canvas, etc.)... Since the release of its version 33, Electron exposes a sharedTextureHandle of windows rendered offscreen, that is actually an IOSurface in macOS and can be shared with main process (node).

For the time being, my implementation:

• Opens a hard-coded window and publishes its content as a Syphon server (no feedback to user).
• Opens an eventually live (eg. received via WebRTC browser-side) canvas to an offscreen window via... local WebRTC... Mmmm...

An intuition based on this issue (but again, I'm taking about things I don't know well, please correct me if I'm totally wrong):

• Syphon Server: we could produce a texture on WebGPU (that looks to be backed by an IOSurface) on Chrome side and send its handle via IPC to publish to a Syphon server.
• Syphon Client: get a Metal texture from Syphon on node side that would be consumed by WebGPU in Chrome.

What do you mean by

in this branch / fork

?