Stateless time APIs

[teh-cmc]

trafficstars

Context

Our 3 SDKs now support our stateful time APIs (although there are still some rough edges), e.g.:

rec.set_time_sequence("frame", 42)
rec.set_time_seconds("sim_time", 632)

This time state is automatically maintained by our thread-local clocks, and automatically injected into the logged data by the RecordingStream's ingestion pipeline. These clocks even keep track of our builtin timelines (log_time, log_tick) without any user intervention. This allows users to fully benefit from our multi-timeline model with minimal effort. Nice.

But, sometimes you want to partially or even fully opt-out from this automatic time tracking.

An example of fully opting out is when ingesting an existing dataset from another database: the data may be dating from years prior and adding a log_time to it doesn't make sense. You want a carbon copy of that original data, no more no less, and for that you need to opt-out of automatic time-tracking and have a way of specifying extra time data.

An example of partially opting out is when you want to associate some extra time data with one single specific log call, in addition to the automatic time. In that case it doesn't make sense to set some global time state just to unset it immediately after the log call. A common example of that is scalars: they use timelines as their X axis. You want to be able to easily custom their X axis and still have them to exist in all other timelines.

Finally, today, we already expose a timeless flag on all our logging APIs. This is effectively a full opt-out mechanism: timeless=true will drop all automatic time metadata when logging. We can use this as a basis for our proposed stateless time APIs.

Proposal

• Keep the timeless flag as-is.

• Add an optional TimePoint parameter to our logging APIs.

  E.g. for python:

  def log(
      entity_path: str,
      entity: AsComponents | Iterable[ComponentBatchLike],
      *extra: AsComponents | Iterable[ComponentBatchLike],
      timeless: bool = False,
      timepoint: TimePoint | None = None,
      recording: RecordingStream | None = None,
      strict: bool | None = None,
  ) -> None
  

  Of course that means we'll also have to expose new APIs to make it possible and user-friendly to build up TimePoints in python.

  Rust:

  pub fn log_with_timeless(
      &self,
      ent_path: impl Into<EntityPath>,
      timeless: bool,
      timepoint: Option<TimePoint>,
      arch: &impl AsComponents,
  ) -> RecordingStreamResult<()>
  

  And something similar for C++.

• Apply the correct behavior depending on both the timeless & timepoint parameters

  The final time metadata for the logged data will always be the union of the time specified in the timepoint parameter and the time from the stateful clocks. In case of conflicts, the values from timepoint overwrites those from the stateful clocks.

  For a partial opt-out behavior, you only need to pass in a timepoint parameter. For a full opt-out behavior, you set timeless=true and pass in a timepoint parameter.

[emilk]

It might make sense to do this first:

• https://github.com/rerun-io/rerun/issues/4080

[teh-cmc]

Update: with send_columns, we are halfway there already, but it would still be very nice to have that access to that kind of expressiveness on the log-oriented APIs.

[thomasjwebb]

This would help me out a lot. I just started using this api and I get the data from another source. I don't need the computer trying to calculate the times. Does anyone have a good example of how I would insert a single row just using send_chunk or send_columns? Are there performance issues with using these methods to just insert single rows?

[Wumpf]

@thomasjwebb send_columns is described in detail on this page https://rerun.io/docs/howto/logging/send-columns Unlike log, send_columns doesn't do any batching, so yes, using it with single rows is quite inefficient while on the other hand the overhead of the automatically added log_time and log_tick is usually not that big. Hope that helps!