Jan Rous

Run ferc_to_sqlite and pudl_etl independent of integration tests

24

This reworks how tests (unit, integration) are done by github actions. In particular, it modifies how ci-integration is executed by running `ferc_to_sqlite` and `pudl_etl` commands independently, making use of dagster...

Consider eliminating the runtime shell wrapper

1

At the moment, some functionality of our pipeline is contained in the wrapper shell script [see here](https://github.com/catalyst-cooperative/pudl/blob/f5cab78c657e9bdfe41c598b0bd50674f851b77a/docker/gcp_pudl_etl.sh#L8). We might want to consider reducing the complexity contained in this shell script...

TESTONLY - sharded ferc_to_sqlite in ci-integration

2

Splits off `ferc_to_sqlite` step into sharded step that generates each dataset independently on a smaller runner. This way we should be able to speed up the process significantly.

The goal of this feature is to develop output differ tool that can compare two outputs and generate human-friendly actionable reports that can help people diagnose the cause of data...