pudl icon indicating copy to clipboard operation
pudl copied to clipboard

Published 20 hours ago • verified publisher icon catalyst-cooperative

TESTONLY - sharded ferc_to_sqlite in ci-integration

Open rousik opened this issue 1 year ago • 2 comments

Splits off ferc_to_sqlite step into sharded step that generates each dataset independently on a smaller runner. This way we should be able to speed up the process significantly.

rousik avatar Dec 13 '23 18:12 rousik

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

review-notebook-app[bot] avatar Dec 13 '23 18:12 review-notebook-app[bot]

This is combination of pre-existing PR that reworks pytest and refactoring of ferc_to_sqlite into separate matrix-based job. I'm aiming to turn this into separate PR when parallel-pytest is merged, but this gives a preview into this proof of concept. I think that it would make sense to move the setup steps (micromamba + other steps) into reusable workflow that can be used here repeatedly now that we have more than just one instance that needs this.

rousik avatar Dec 14 '23 18:12 rousik