Support multiple sinks at once (`tee` logging)

[teh-cmc]

Something along the lines of this pseudo code:

struct LogSinkBundle(Vec<Box<dyn LogSink>>)

impl LogSink for LogSinkBundle { ... }

also need to update CLI args helpers for both Rust and Python to support mixing --save with anything else.

[emilk]

It is very useful to be able to dynamically add and remove sinks at runtime.

For instance:

• you start rerun with no sinks
• something happens, so you now start dumping to an .rrd on disk
• you hit a rare corner case happens, so you call spawn and start streaming to the viewer, then stop that stream once you reached the end of the corner case

This mean we need an API along the following lines:

rerun = rr.new_recording(app_id=…, …)
rerun.add_sink(file_path="foo.rrd")
viewer_sink_id = rerun.add_sink(spawn=True)
…
rerun.remove_sink(viewer_sink_id)

When no sinks are attached, rerun.log(…) should be a no-op

[emilk]

An interesting case to consider is streaming to an .rrd and calling spawn. We could set up the native viewer to be able to stream-load an .rrd file that is being written to, requiring no extra WebSocket connection. That would also mean the rerun library without the WebSocket feature (and huge dependency tree) could still support spawn (see https://github.com/rerun-io/rerun/issues/4788)

• https://github.com/rerun-io/rerun/issues/4056

[vmayoral]

Adding to the thread another potential use case of interest: https://github.com/rerun-io/rerun/issues/6685

In a nutshell: Periodic calls to rr.save() to generate smaller/manageable log files. This is particular relevant in robotic systems generating tons of data from depth sensors and multiple cameras.

[vmayoral]

Hello all, any updates on this topic? Rationale at https://github.com/rerun-io/rerun/issues/1998#issuecomment-1887409983 makes a lot of sense to me as well. Thanks!

[spewil]

Tracking this as well, having the ability to tee at the SDK would be great! I'd like to be able to persist logs as well as "check" on them as I wish. If I can help push this along, let me know!

[bjin-bdai]

Came across this as well while trying to simultaneously stream data to the viewer while log to disk. I think this is a pretty strong feature so that we can safely have our data on disk in case the viewer crashes mid-session before we're able to save anything.

Saving the data without being able to view certain data live limits some use-cases for this library. Especially when we want to use this for longer sessions

[jprochazk]

To address one part of this issue (stream to gRPC and save to a file at the same time), we're adding a set_sinks API to our SDKs in 0.24

[therishidesai]

To address one part of this issue (stream to gRPC and save to a file at the same time), we're adding a set_sinks API to our SDKs in 0.24

Are there plans to also support the GrpcServerSink? Right now it seems like I can only get it to connect to a grpc server and send to a file

[jprochazk]

Are there plans to also support the GrpcServerSink? Right now it seems like I can only get it to connect to a grpc server and send to a file

Yes, the plan is to eventually expose all sinks in a first-class way.