WIP: Xbrl integration

[zschira]

This PR tracks the integration of FERC XBRL data into PUDL. Specifically, it's focused on updates to the ferc1_to_sqlite.py script to generate raw SQLite DB from XBRL data, and updating the extract step to work with this data. Developing transformations to use XBRL data with historical data is out of scope of this PR, and still forward work.

Criteria for merging

Merge other PRs into this branch

• [ ] are tests passing on the subset of table(s) that have been migrating?

Merge this branch into dev

• [ ] all existing pudl tables are integrated into this branch and tests and passing

[zaneselvans]

Hey @zschira it seems like the CI is failing here because it's not finding the FERC 1 DB, as the pudl_settings['ferc1_db'] entry is now points at the XBRL version of the DB. It seems like we'll now need to have two different ferrc1 DBs with different references for each of them that can be selected appropriately in the FERC 1 extract step.

[codecov[bot]]

Codecov Report

Base: 83.3% // Head: 85.1% // Increases project coverage by +1.8% :tada:

Coverage data is based on head (801223b) compared to base (006382e). Patch coverage: 92.7% of modified lines in pull request are covered.

:exclamation: Current head 801223b differs from pull request most recent head d979668. Consider uploading reports for the commit d979668 to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##             dev   #1665     +/-   ##
=======================================
+ Coverage   83.3%   85.1%   +1.8%     
=======================================
  Files         67      72      +5     
  Lines       7308    8183    +875     
=======================================
+ Hits        6090    6968    +878     
+ Misses      1218    1215      -3     

Impacted Files	Coverage Δ
src/pudl/metadata/fields.py	100.0% <ø> (ø)
src/pudl/metadata/resources/ferc1.py	100.0% <ø> (ø)
src/pudl/metadata/sources.py	100.0% <ø> (ø)
src/pudl/transform/ferc1.py	94.7% <ø> (+2.7%)	:arrow_up:
src/pudl/transform/params/ferc1.py	100.0% <ø> (ø)
src/pudl/cli.py	69.6% <33.3%> (+8.7%)	:arrow_up:
src/pudl/convert/ferc_to_sqlite.py	58.9% <44.4%> (ø)
src/pudl/analysis/service_territory.py	28.1% <50.0%> (-0.9%)	:arrow_down:
src/pudl/convert/censusdp1tract_to_sqlite.py	72.3% <50.0%> (+6.8%)	:arrow_up:
src/pudl/convert/datasette_metadata_to_yml.py	66.6% <50.0%> (-33.4%)	:arrow_down:
... and 46 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[review-notebook-app[bot]]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB