Feature request: include other UIView content in the recording, on top of ARSCNView

[okvivi]

First, great work with this library - really really useful!

In the app that I am building I am overlaying 2d content (UIViews) on top of the AR content.

When recording, I would love to be able to record photo/video with everything that's in this other view that's overlaying the ARSCNView, not just the ARSCNView content.

Any thoughts on whether / how / when that would be possible?

Thanks! Octavian.

[AFathi]

@okvivi,

Since you're using ARSCNView, I believe you can easily diffuse the UIView layer into your scene in order to record a video with the 2D content overlay.

To diffuse the material of a UIView into an ARSCNView try implementing the following block of code:

            // UIView overlay
            let myView:UIView!

            // ARSCNView
            let scnView:ARSCNView!

            // Create a SCNMaterial and diffuse [myView] into its contents
            let material = SCNMaterial()
            material.diffuse.contents = myView.layer
            
            let plane = SCNPlane(width: myView.bounds.width, height: myView.bounds.height)
            let node = SCNNode(geometry: plane)
            node.geometry?.firstMaterial = material
            node.position = SCNVector3Make(0, 0, 0)
            
            scnView.scene.rootNode.addChildNode(node)

Try this out and let me know if that's what you're trying to achieve.

-Ahmed

[okvivi]

Hey, thanks for the super quick reply! That's not quite what I am trying to achieve, though I might try a similar hack at some point...

The UIView that I'm trying to render in the video is like a 2D layer on the screen - a 2D drawing that's always on the user's screen.

Like in SnapChat, where you have the 2D overlay filters that show your location, or something... just an image on top of the camera, in 2D not in 3D. As the camera moves, this UIView remains like an always-on sticker on the camera.

Does that make more sense?

[AFathi]

@okvivi, are you trying to merge the UIView layer after recording the video?

If so, your best option would be using AVMutableComposition. Otherwise, the approach I suggested above will still work!

[okvivi]

Hey Ahmed -

The solution you're suggesting does not work. Even conceptually, showing a plane that shows a UIView in it is not what I'm looking for.

The solution I'm talking about is more like this one here https://github.com/lacyrhoades/SceneKit2Video/blob/master/Classes/VideoRenderer.swift#L134

I want to overlay a static, 2D image, screen-sized, onto the video / photos that ARVideoKit is taking.

Anyways, sounds like right now that's not possible. Consider this a feature request. :-)

[AFathi]

@okvivi, that makes more sense! I misunderstood the question.

Will consider adding this feature in the next update!

[okvivi]

For what it's worth, in case it helps anybody in the future.

If you're using SCNRenderer to grab video from a SceneKit (ARKit) scene, you can use renderer.overlaySKScene to overlay a SpriteKit scene that renders 2d content.

That way the recording is really effective and fast, and you can do whatever 2D animations and overlays you want, and the SCNRenderer takes care of the blending.