listen on github events instead of polling

[TomasTomecek]

Release bot polls the tracked github project every 10 minutes in current implementation. It would be much more efficient if it was able to receive Github's callbacks. Alternatively, it can listen on fedmsg (but that requires for the tracked projects to be included in github2fedmsg).

Work to be done:

• Discuss the best solution with the upstream maintainers (that's us)
• Implement it - release bot is actively able to react to events

Ultimately, our plan is to:

• Deploy release bot to openshift online as a github app.
• Add it to all our upstream projects.
• The bot will listen to all the events via github's callbacks.
• And we can have only a single deployment for all our projects.

This is a very good candidate for Google Summer of Code or Red Hat Open Source Contest.

[marusinm]

I will implement callbacks processing from Github webhooks and deploy release bot on Openshift online. This webhooks handler should be configurable in a conf.yaml or replace refresh_interval if r-bot runs with public domain.

Later on, when Github app or single deployment for more projects mentioned in this issue will be implemented, the handler for GitHub events will be prepared.

[TomasTomecek]

Awesome, can't wait to have this one implemented.

I have a few suggestions:

• please think about the implementation and ideally share the design with us before starting hacking on it - a list of proposed new dependencies, new classes, how the code will be organized etc. the more details the better
• when you start hacking on this, it would be very nice if you implemented things step by step and NOT have one colossal PR with all the changes: more PRs with smaller changes are much easier to digest
• please do a brief research on how people already solve this issue (accepting github webhooks), I wouldn't be too surprised if there was a python library for this already
• and have fun!

[marusinm]

Current state: I have deployed release-bot in OpenShift online. I created new webhook on repo which is watched by this release-bot instance. Then I built simple flask based application and I was able to listen to all newly created Issues and Pull requests. Everything is working in Openshift online.

please do a brief research on how people already solve this issue (accepting github webhooks), I wouldn't be too surprised if there was a python library for this already



I also find some nice libs about how to process webhook callbacks:

• Flask based micro-framework for Github webhooks - I prefer this one
• more coplex Django based framework for webhooks (not only Github webhooks)

Ideas to discuss: 
I propose add new option webhook_callback_url in conf.yaml. Create a module (class) which will process Github callbacks using mentioned lib. This may produce two scenarios:

• refresh_interval is configured in conf.yaml - release bot works in a normal way with infinity while loop and refresh_interval.
• Or webhook_callback_url is configured in conf.yaml - release bot will listen for callbacks instead infinity loop, especially when running in Openshift or as Github app in the future. 


This could be first iteration of the task.

Note

• Above scenarios are based on webhooks which user have to set manually. It should be possible to create webhook on repo via GitHub API through release bot. We can create new issue for that.

[TomasTomecek]

+1, using django for this would be a total overkill, flask will be just fine

I checked the proposed flask library and personally -1 on using it:

• Latest commit e9a70dd on Sep 7, 2017, so it's no longer maintained, pretty much.
• The code has 150 lines only, and since we'd be interested in releases, issues and PRs only, we can easily have the code for processing the webhooks directly in our bot.


I propose add new option webhook_callback_url in conf.yaml.

Care to elaborate? Why do we need to make this configurable?

I agree that we should distinguish between the two modes of release bot via configuration.

Thanks Marek, very nice progress!

[marusinm]

we can easily have the code for processing the webhooks directly in our bot.

+1, Thanks for checking the library. I have already tried to write some handler for such webhooks, it shouldn't be a problem.

Why do we need to make this configurable? I agree that we should distinguish between the two modes of release bot via configuration.

I was thinking to use webhook_callback_url to distinguish the two modes of release-bot and also we will be able to use url to set webhook in bot instead of bothering a user in the future. However, now I realized if we will create Github app in the future there may be better solution for this.

Can we just add option webook_handler inside conf.yaml with true/false values instead of my previous proposal with webhook_callback_url? Or just tell me if you have better idea.

[shresthagrawal]

@marusinm Hey the Idea looks pretty cool! Is their any part of the issue that I can work on, just to speed up the process?

[marusinm]

@shresthagrawal thank you I appreciate this but I am almost finished. Maybe another time.