rerun icon indicating copy to clipboard operation
rerun copied to clipboard

Add .rrd files to `git lfs` to test backwards compatibility

Open emilk opened this issue 10 months ago • 1 comments

We want to have automated tests to make sure we have backwards compatible chunk loaders.

Ideally we should add .rrd files to git lfs for all previous versions of all components and archetypes.

What to test?

  • They load without error/warning
  • Passes rerun rrd verify
    • https://github.com/rerun-io/rerun/pull/9128
  • Produce the same rerun print output?
    • Maybe just the column headers (names, meta-data)?
    • That's a useful sanity check when we update the .rrd:s in the test set, that the new ones covers approximately the same things
  • Produce the same visuals? Difficult to get right

What .rrd files?

  • [ ] Snippets?
  • [ ] Examples?
  • [ ] One .rrd file containing all known components/archetypes/datatypes?

When and how do we add more .rrd files to the test set?

What if something does break?

We do not yet promise full backwards compatibility, but any CI failure regarding this should be a red flag, and we should only accept it (and upload a new .rrd file) if we are really ok with this breaking change.

What remains to do?

  • [ ] Decide on exactly what .rrd files to test
  • [ ] Automatically verify that all snippets and examples etc have .rrd:s that are checked in
  • [ ] Add .rrds under folders with version string
  • [ ] Compare all versions of old .rrd files (e.g. from 0.22, 0.23, 0.24…)
  • [ ] Ensure we add new versions of snippet and examples iff needed
  • [ ] Improve the script for updating said .rrd files to:
    • [ ] Not be a bash script (-> Python or Rust)
    • [ ] Add missing files (e.g. when adding new snippets) without changing the existing files
    • [ ] Update all existing files (when intentionally breaking backwards compatibility)
  • https://github.com/rerun-io/rerun/pull/9370

Potential improvements

  • [ ] Migrate the checked in snippet rrd files, then compare them to the output of the latest snippet

emilk avatar Feb 22 '25 10:02 emilk

compare_snippet_output.py should compare its output with the .rrd files checked in to git lfs:

  • If missing, it is an error (a new snippet needs to check in an .rrd for its output)
  • If it differs, it is an error

Complications

If we change a snippet (so that it outputs something different), we need to either:

  • Update the file on git lfs
  • Add a new file to git lfs alongside the old one
  • Add an opt-out flag for comparing to the file on git lfs to snippets.toml

Since this should be a rather rare occurrence (our snippets tend to stay pretty stable), any one solution will do.

emilk avatar Apr 15 '25 11:04 emilk