Auto revert broken trunk commits

[huydhn]

The number of force merges is on the raise recently and it's proportional to the increase in red trunk commits. In addition, we also had a CI sev in Feb where viable/strict wasn't updated for 5+ days https://github.com/pytorch/pytorch/issues/120368. In both cases, handling broken trunk failures wasn't done in a timely manner. There are also some CI jobs where oncall might feel reluctant to revert right away without checking with the owner, for example rocm and inductor jobs.

The proposal here is to let the bot do it instead:

1. The bot will post a comment if it considers a commit is breaking trunk and should be reverted. It's cc the author and PyTorch Dev Infra (oncall)
2. If the revert is not needed, the author or oncall can reply to tell it to not do so, i.e. https://github.com/pytorch/ci-infra/pull/45#issuecomment-1961716433
3. Otherwise, after waiting for X hours, the bot will revert the change
4. [Optional] If the revert doesn't help and similar failures still occur in trunk, the bot could reland the reverted change providing that the necessary approval is still there.

cc @clee2000 @kit1980 @PaliC @malfet @seemethere

[huydhn]

Some further thoughts from the team:

• We should clearly define the condition when the bot could do the revert:
  • Not a CI sev
  • Not an upstream failures, i.e. conda
• It's a good idea to cap the limit of reverts the bot could do within a period of time

[huydhn]

AI: Add a filter to HUD to zoom in the streak of failures on HUD that is associated with the reverts