Refactor to improve synchronization and testability

[bryanburgers]

A start to the task of refactoring to improve synchronization and add more testing to Homu. More information about the target end goal can be found at the Proof-of-Concept PR #44

• [x] Test command parsing logic (#32)
• [x] Test authorization code
• [x] Pull PullReqState into its own file
• [x] Use Timeline events to perform the initial synchronization
• [x] Add tests for timeline event synchronization
• [x] Track state for multiple tries
• [ ] Migrate webhook handling code to use same API as initial synchronization

[bryanburgers]

Making progress! I now have Homu synchronizing its initial state from GitHub Timeline Events in a pretty accurate way. A few differences yet where I need to track down what the actual expected state should be.

Current Homu:

Local Homu with these changes:

Anybody can pull this branch and try it locally. Some notes about that:

• It only does the synchronization. Because it won't be set up for webhooks, it will not keep in sync at this point.
• You can use any access token from any user. I'd suggest not using an access token for a user that has access to rust-lang/rust, just in case there's a bug somewhere that would affect the real functioning of bors.

[bryanburgers]

Status update

I've been running a "follower" against the rust-lang/rust repo using this new sync method for a couple of days now. Every minute or so, it grabs all of the PRs from the API and resyncs them from the previous sync point to current.

https://homu.burgers.io/queue/rust

Takeaways:

• I'm frequently seeing it stay in sync! Which means that this method is definitely working as expected.
• This has actually been a very effective way to find edge cases that I wasn't aware of. Whenever the follower doesn't match the current version (https://buildbot2.rust-lang.org/homu/queue/rust), I figure out why and adjust the synchronization code.
  • Unfortunately, that won't work so well when I need to start reacting to changes and issuing comments, at which point I'll need to drop back down to test repos
• current version often doesn't have ALL the open PRs. It appears that on initial sync, it might only synchronize the newest 100 open pull requests (?)
  • This doesn't seem to matter much in practice. Old PRs end up showing up after they get updated anyway.
• I often see differences in "Mergeable" and "Assignee" columns. It looks like current version doesn't always stay in sync for those fields
• Rate limit: GitHub v4 has a 5000 "cost" rate limit. Despite pulling changes every minute, I've only ever used 398 of those (still have 4602 remaining) before the rate limit resets. So this method doesn't appear like it will run into issues running up against the rate limit.

[Mark-Simulacrum]

I think GitHub returns at most 100 objects from any query, so we're probably not asking for the next page on some query.

[bryanburgers]

@Mark-Simulacrum

https://github.com/rust-lang/homu/blob/abd0083ba6a738e0bc1a1bc69e64676ff9032ebf/homu/main.py#L1495

Right. https://github3py.readthedocs.io/en/stable-0.9/repos.html#github3.repos.repo.Repository.iter_pulls suggests that it returns all available pull requests, but it seems likely we're just getting the first page of results.

But at this point, I don't believe it's worth investigating.

[Mark-Simulacrum]

Somewhat agreed, though I think we'd want to investigate before going forward -- missing PRs in the queue are annoying because you can't easily tell if we have all of them by comparing the number, you have to check one by one.

[bryanburgers]

Status update: I've been incredibly busy the last couple of weeks with other things and haven't been able to get back to working on this.

From what I see, this change is not something I can introduce in pieces, so it will take time to get it to a production state.

I hope to be able to return to this mid-August.

[bryanburgers]

Slowly but surely trying to keep working on this.

I now integrated a history of tries and builds, based on the GitHub history and bors' past comments, so we can have something like the following image (but with more UI work).

https://homu.burgers.io/results/rust/58281

I'll keep running https://homu.burgers.io/queue/rust while I work on this. From what I can tell checking in every once in a while, it's a pretty accurate representation of the state of the world. And is almost always more accurate with respect to mergeability and assignees.

Unfortunately, it's getting long and is going to be a huge pain to review, and a huge risk to switch over.