etl
etl copied to clipboard
Automate PR creation when updating steps
Our guide for updating data suggests creating reference and review branches as best practice. We already use etl d draft-pr
command which has proven to be handy for automatic PR creation. We could do similar automation for updating data:
- Create reference branch
- Update steps (by using functions from ETL dashboard)
- Commit new steps and push reference branch to Github
- Create a review branch with PR
@lucasrodes suggested
Maybe, we want to add that when updating the steps in ETL Dashboard? E.g a checkbox or something ‘create reference and review branches’. Title and name of the branch could be dynamically generated based on the datasets selected for update (edited)
@pabloarosado suggested
I don't think all that can easily be done automatically (because step 2. can get a bit tricky). But maybe there's a better way around it.
It'd make sense to at least create reference & review branches automatically, and then see whether it's worth automating the rest. Leveraging etl d draft-pr
could make it relatively easy.