Zach Schira

WIP: Publish raw FERC XBRL DB's to datasette

1

This PR follows issue #1830 for adding the ability to publish all raw FERC XBRL based SQLite databases to datasette. ## Tasks ### Create SQLite databases from PUDL - [...

WIP: Xbrl integration

2

This PR tracks the integration of FERC XBRL data into PUDL. Specifically, it's focused on updates to the `ferc1_to_sqlite.py` script to generate raw SQLite DB from XBRL data, and updating...

Create infrastructure for publishing all raw FERC SQLite DB's extracted from XBRL data

4

Our [FERC XBRL Extractor](https://github.com/catalyst-cooperative/ferc-xbrl-extractor) works with FERC Froms 1, 2, 6, 60, and 714. Form 1 is the most well integrated with PUDL, but even then many tables have not...

Integrate FERC XBRL data into PUDL

13

# Background After creating tools to translate the FERC XBRL filings into SQLite databases, we decided that the old Visual FoxPro DBF and new XBRL data will need their own...

# Background This pull request adds FERC Forms 6 and 60 datasources. It does not integrate these forms into PUDL in any way, but it will enable us to archive...

Update `ferc1_to_sqlite` to migrate XBRL data to a second "raw" FERC 1 SQLite DB.

3

## Background The first step to integrating XBRL data into PUDL is being able to generate the raw SQLite DB from archived XBRL filings. The `ferc1_to_sqlite` script can be updated...

Integrate the new FERC 1 XBRL archive into the PUDL Datastore.

1

## Background In order to ingest XBRL data into PUDL, we need a datastore that can interpret XBRL archives (#1593). The archives consist of a set of XBRL filings, and...

Identify common mutations in FERC steam plants data

1

### Purpose We want to settle on a final validation metric to feel "good enough" about merging the FERC-FERC plant matching refactor (#3137) into PUDL. Because there is no ground...

How to handle storage/access to pre-trained model weights

12

## Background The updated FERC-FERC inter-year plant matching model in #3007 uses PCA, which is much faster if we pre-fit the model and save the weights somewhere. However, when we...

## Background We have all of the SEC filings available in GCS with a metadata DB. To aid exploratory extraction of SEC filings, we need tooling to work with this...