Api for saving screenshots and animations

[chisarie]

Hello!

It would be great if there was a way to log a pointcloud or mesh to rerun, specify a camera pose, and programmatically save a "screenshot" of that object from that specified pose. It would be very useful for evaluating and comparing different things, and for publishing purposes.

An extension of this would be to provide a list of camera poses, and use it to generate multiple screenshots that can be then stitched together to make a video or gif.

From looking at the docs, it seems this functionality is not there yet, is it?

Thank you in advance for the help!

[nikolausWest]

Hey @chisarie! Thanks for the question / feature request. Screenshots from the command line actually just landed (#2293). The ability to construct the Rerun Viewer from a declarative spec is not there yet, but under way (see PR with RFC here: https://github.com/rerun-io/rerun/pull/2302). Once that lands with the ability to also set a 3D view's eye I believe most of what you want should be there

[chisarie]

Sounds great! One small thing to note which is different than https://github.com/rerun-io/rerun/pull/2293 is that I'd like to have a "screenshot" of a viewport alone, without all the gui around it. Looking forward to it :)

[chisarie]

Maybe screenshot is not the right word for this, what I mean is more like an offscreen renderer, similar to Open3d's and Pyrender's ones: http://www.open3d.org/docs/latest/python_api/open3d.visualization.rendering.OffscreenRenderer.html, https://pyrender.readthedocs.io/en/latest/examples/offscreen.html

[nikolausWest]

Sure thing. We have the pieces in place to screenshot single views without UI surroundings as well. I'm not familiar enough with the offscreen render in Open3D to know how that works exactly but if you also need the rendering to run headless that would be a bit further off with Rerun.

Would you mind sharing a bit more background around how and why you would intend to use a feature like that?

[chisarie]

Mainly to generate comparison pictures to use on publications paper and website. For example to compare the outputs of different methods. If I have to take all screenshots manually it would be very time consuming and also not precise (camera position should always be the same to compare properly). To use rerun for this, another important feature would also be to be able to set a white background, see https://github.com/rerun-io/rerun/issues/754

[nikolausWest]

Thanks @chisarie, that makes a lot of sense!

[kuddai]

Hi, I have saw that for internal development testing there are an offline testing capabilities/like offline/headless 3d rendering? Can you please tell a little bit more how does this work entirely? Curious because while we are waiting for official api support, I want to take screenshots in my fork.

[Wumpf]

@kuddai In a nutshell we're using egui_kittest together with an internal test harness that mimics key part of the viewer.

The main problem here is definitely not taking screenshot itself (we have this from the ui! both for individual views & the viewer itself) and even headlessness should be fairly easy to achieve. Instead, the main hurdle for this feature is wiring it all up into an api that communicates on a non-blueprint non-recording stream communication channel from sdk to viewer (and back ideally). This is something we also need to support callbacks (which right now we only support for python notebooks and Rust processes embedding the viewer)

-> I think a relatively easy first step would be to expose a screenshot method in the Viewers "in-process" Rust API, similar to Rust callbacks (see example for those)

[iwanders]

I had similar needs and found that if you are okay with a minimal amount of interactivity you can do the following: Open the viewer and double-click a pinhole camera in a 3d view. This will make the camera in that view snap to the pinhole's view & location. If that pinhole camera's Transform3D then changes, the view keeps moves with it. To hide the gray frustrum visualisation set the pinhole camera's line_width to zero in the selection panel on the right. This does give you programmatic control over the camera pose. By calculating the camera pose and updating it each frame one can create nice camera animations (I just recorded my screen to capture the video).