rerun-io
MVP for Audio Playback (converted to project)
This issue has been converted to a project MVP for Audio Playback
Context
There's a much more detailed request for a broader set of audio features here: https://github.com/rerun-io/rerun/issues/2852
This issue is about a more minimal feature to just get audio playback working in the most minimal capacity.
Proposal
Make it possible to log audio assets as a single (or small number of) blobs and then have these play back only during "play" mode in the viewer. Avoid the hard questions around hings like audio-packetization, time-spans, and cross-timeline synchronization.
Simplifying assumptions:
- The timestamp of the audio asset is the time at which audio should start playing on the given timeline.
- The audio file should only be logged to timelines where the time scale matches the native audio time scale of the file.
- Only support playing back one audio stream at a time.
- Only play back audio when specifically in "play" mode with a 1.0 time-scale.
Suggested implementation
- Add the ability to log a binary blob containing audio data in a small number of common formats using a similar approach to
Asset3D. - Create an Audio "space view" which only supports a single audio asset. The view probably just shows something like an audio icon and maybe a volume / mute control.
- When scrubbing time, the view will load the approach "LatestAt" audio payload given the current time selection, and then offset the stream to the correct relative timepoint.
- When time is "playing" the view will playback the audio file using the native clock and track to make sure the audio clock and the native clock continue to be aligned.
Crates
Thank you @jleibs for discussing this and taking time out to create the issue. This level of audio support is going to help us get unblocked for our project and really centralize ReRun as "the" visualizer.
I created a small proof-of-concept branch experimenting with this a while back. It is not mergable, but may be inspirational for when we decide to work on this for real:
https://github.com/rerun-io/rerun/compare/main...emilk/audio-poc
@Relento no, nothing new yet. Other things did get prioritized over this after all
