etl icon indicating copy to clipboard operation
etl copied to clipboard

Automate PR creation when updating steps

Open Marigold opened this issue 7 months ago • 3 comments

Our guide for updating data suggests creating reference and review branches as best practice. We already use etl d draft-pr command which has proven to be handy for automatic PR creation. We could do similar automation for updating data:

  1. Create reference branch
  2. Update steps (by using functions from ETL dashboard)
  3. Commit new steps and push reference branch to Github
  4. Create a review branch with PR

@lucasrodes suggested

Maybe, we want to add that when updating the steps in ETL Dashboard? E.g a checkbox or something ‘create reference and review branches’. Title and name of the branch could be dynamically generated based on the datasets selected for update (edited)

@pabloarosado suggested

I don't think all that can easily be done automatically (because step 2. can get a bit tricky). But maybe there's a better way around it.

It'd make sense to at least create reference & review branches automatically, and then see whether it's worth automating the rest. Leveraging etl d draft-pr could make it relatively easy.

Marigold avatar Jul 23 '24 07:07 Marigold