Trigger CI builds for pytorch/pytorch and domain libs in pytorch/builder via inter repository dispatch

[langong347]

This is the first step to synchronize and consolidate pytorch repository builds in the builder repo:

• pytorch/builder will send a dispatch event to one of pytorch/pytorch, pytorch/vision, pytorch/test, pytorch/audio
• Each repository will listen on the dispatch event to trigger their own CI build

Status:

1. Enabled cross repo triggering of binary builds from pytorch/builder to pytorch/pytorch through CircleCI API:
   • Pull request: https://github.com/pytorch/builder/pull/874
   • Triggering workflow on master branch of pytorcy/pytorch needs to resolve concurrency issue.
   • Collect results from pytorch/pytorch and display inside pytorch/builder?
   • Should we roll back #860 for binary builds enabled previously as both aim at testing changes in pytorch/builder from the workflow of a dependent repo? Or we carve out binary builds to migrate standalone into pytorch/builder?
2. Triggering through GHA is blocked at this moment due to limitation of Github API
   • Approach 1: reusable workflow (concurrency is not supported in called workflow as in the case of binary build, example GHA run)
   • Approach 2: repository dispatch event trigger (only the default branch of the dependent repo can be dispatched onto which means we won’t be able to use a test branch in pytorch/pytorch to test the triggering, see ref)
   • Pull request: #877

Parent issue tracker: https://github.com/pytorch/pytorch/issues/66656

[malfet]

It sounds like prerequisite for this one would be to move existing CI workflow to GitHub action (I'm not sure if CircleCI supports repository dispatch)

Also, can you please elaborate in what cases it will be necessary?

[langong347]

Using Github Action is one option. We can explore other options (like probot) to do the same inter-repository trigger.

Regarding the motivation: This is to have an e2e CI build for strongly connected pytorch libraries to early detect error due to changes in one of them. Using repository_dispatch or similar idea is to link the builds together similar to what we have in a mono-repo.

[orionr]

@zhouzhuojie I think you had some thoughts about this in that we could build this system out and then incrementally use for GHA on new repos (torchonnx, for example) as well as migrating existing domain libraries over time