rerun icon indicating copy to clipboard operation
rerun copied to clipboard

Keep `torch` dependency from bloating our common pixi environments

Open jleibs opened this issue 1 year ago • 1 comments

Even though pixi is fairly efficient when it comes to downloading, caching, etc. the torch dep still adds several gigs to the environment.

This has two notable consequences:

  • It actually causes smaller github runners to fail because they run out of storage. (https://github.com/rerun-io/rerun/pull/6964)
  • It causes significantly more churn on the github cache of the pixi environments since we blow the 10GB limit with just a few environments.

We currently need torch for 2 things:

  • Running unitests that depend on the torch tensor type
  • Running a handful of examples that depend on torch

Proposal

  • Make the torch dep for unit-tests optional. Only include torch arrays if the dep is present.
    • If we're feeling particularly fancy we could fallback-import a torch tensor shim, but I'm not sure it's worth the effort.
  • Split out the torch environment dep so we end up with wheel-test and wheel-test-torch
  • Remove any examples that need torch from main
  • Only use wheel-test-torch on nightly to:
    • Validate the unittests still pass with torch
    • Run any examples that depend on torch
  • Disable saving the wheel-test-torch cache. Since we only run it nightly it's not worth causing cache churn for it.

jleibs avatar Jul 23 '24 12:07 jleibs

Maybe things depending on torch should be in its own repo?

  • https://github.com/rerun-io/rerun/issues/5661

emilk avatar Aug 05 '24 14:08 emilk