functorch icon indicating copy to clipboard operation
functorch copied to clipboard

Set up automatic sync with fbcode, stricter CI requirements, and disable pushing to mater.

Open Chillee opened this issue 2 years ago • 2 comments

Since more projects are starting to depend on Functorch, we may want to establish more reliable infra/CI, including:

  1. Automatic sync with internal functorch.
  2. Stricter CI expectations (not totally sure what should be expected here).
  3. Perhaps disable pushing to master directly.

cc: @zou3519 @samdow @gmagogsfm

Chillee avatar May 12 '22 01:05 Chillee

Moving on here since we started some internal discussion but it might get lost

Automatic sync with internal functorch.

This seems difficult since nightly != internal master != what's on difftrain. Specifically, a fix that fixes difftrain might break functorch's CI for ~a day? Then we're losing signal for a day

IIRC there were also some issues from the devinfra team about us trying to jump on the difftrain

samdow avatar May 13 '22 21:05 samdow

The requirements we should consider are:

  • developer velocity (for functorch devs)
  • developer velocity (for pytorch/pytorch devs)
  • internal customers
  • limiting internal breakages
  • The ability to release functorch out-of-tree (and more often than PyTorch)

Are there any more we should consider?

There are a couple of options: (deeper writeup in https://docs.google.com/spreadsheets/d/1J9gXj71AXNgh7Yr_CNcjsYgZqsLwowE_Eg8BFE77f2E/edit#gid=0)

Option 1: upstream functorch

Copy-paste functorch into pytorch/pytorch

  • Pros: No more sync issues.
  • Cons: Need to do some degree of convincing and/or code cleanup.

Option 2: keep functorch out-of-tree (in the short-to-medium-term)

This requires a bunch of infra changes (what @samdow was mentioning):

  • functorch CI has to develop off of PyTorch master instead of PyTorch nightlies
  • GHF difftrain needs to somehow sync both PyTorch master and functorch into fbcode. That is, not all commits in PyTorch master will make it to fbcode immediately so someone needs to be there to figure out what commits of functorch should make it into fbcode.

Pros: stops the pain Cons: Requires coordination and infra changes

zou3519 avatar May 16 '22 15:05 zou3519