open-grid-emissions
open-grid-emissions copied to clipboard
singularity-energy
Re-assess cleaning of EIA-930 data
Physics-based cleaning
We use gridemissions to clean EIA 930 data. Currently we use 930 generation, however, this causes issues during the calculation of consumed emission rates. (see #214)
A better solution may be to do physics-based cleaning using our own generated MWh. This would require modifying the optimization algorithm to prioritize changing interchange and demand values over net generation values, since we trust our net generation estimates over the 930 values.
We would also need to provide user guidance on compiling cvxpy, since it can be extremely slow for some users and this change would require running optimization during every run of data_pipeline.py.
Systematic issues
The initial correction of systematic issues in 930 data depended on comparison of interchanges between BAs. Now that we have the OGE dataset, we can do a more thorough comparison.
One example of this is AZPS, where there is a sign flip on June 1, 2020, that was not caught in the initial analysis.
Red is our `net_generation_mwh`, blue is total interchange as reported to EIA-930 after `gridemissions` cleaning
So for this specific issue with AZPS, it looks like in the raw data, the AZPS-SRP interchange flips from positive (net exporter to SRP) to negative (net importer from SRP) on Jun 1, 2020.
However, SRP reports being a net exporter to AZPS consistently across this period
Thus I think that AZPS had been incorrectly reporting its imports from SRP prior to 6/1/2020 as exports. They fixed it going forward, but never corrected the historical data. To fix this issue we would want to flip the sign of the AZPS-SRP interchange prior to this date.
But when did this AZPS-SRP issue start? It seems that SRP consistently reports itself as a net exporter to AZPS in the 2,000-4,000 MWh range back through 2019. However, the AZPS numbers get weird. Instead of a sudden sign flip like we saw in June 2020, it seems like they slowly start to drift from reporting net imports to net exports in November 2019:
However, the absolute magnitude of these numbers also doesn't match the magnitude of the exports reported by SRP for this same period:
It is almost as if the data were inverted and scaled somehow. We can see this trend of the wrong magnitude continue all the way to June 2020, when the sign flips (not only does the sign flip, but it goes from +1000 to -3000, correcting both the magnitude and the sign)
It almost appears that if you subtracted ~3000 from the SRP numbers, then flipped it, you would get approximately what AZPS was reporting...
So after playing around with these timeseries, I cannot identify a clear/consistent relationship between the two timeseries. It might just make sense to invert the SRP-reported interchange with AZPS to fill this timeseries from AZPS.