gridstatus icon indicating copy to clipboard operation
gridstatus copied to clipboard

Improving status info in ISONE, SPP, & ERCOT interconnection queues

Open TrentonBush opened this issue 1 year ago • 2 comments

Hello 👋🏼I work for Catalyst Cooperative and am starting to use GridStatus to pull ISO interconnection queues. Thanks so much for your work building this functionality and publishing it as open source!

While exploring the queue data, I ran into a couple of minor issues. I'm happy to submit a PR in the next week or two to address them, but wanted to outline them here in case somebody else is already working on any of them.

  • [x] In SPP, GridStatus currently overwrites more detailed status information with simplified categories. Simple categories are useful, but preserving the detailed information is also useful for tracking project progress. I think this is as simple as making a new column to preserve the original values.
  • [x] ISONE publishes their interconnection queue as an HTML table (great!) but they encode queue status information as graphical icons (less great), with the category label as the title text in each <img> tag. GridStatus does not currently parse this information, so it is all lost. It should be pretty easy to fix by, for example, replacing the <img> tags with their title text prior to pd.read_html().
  • [ ] Also in ISONE, GridStatus defines Queue ID with what the original data calls Queue Position. Queue Position is only unique for active projects. It has a 1:m relationship to both withdrawn and completed projects. IMHO considering the Queue ID is unique in all the other ISOs (excepting a tiny number of errors in the source data), it is a bit sacrilegious to label a non-unique value with the sacred letters "ID". Unfortunately I don't see a natural key for those items -- perhaps we can come up with a surrogate key?
  • [ ] I think you already call this out in #76, but to be more explicit: when comparing ISOs, status category labels that are superficially synonymous can actually differ considerably in their meaning. For example, in MISO & PJM queues, the term "active" refers to projects up to the point of IA execution, whereas in CAISO, ERCOT, ISONE, & NYISO queues "active" applies up to the point of commercial operation, and the sickos at SPP call every project that wasn't withdrawn "active", including operational facilities. There is often a year+ between IA execution and commercial operation, during which construction, energization, and finally synchronization occur. I think it would be useful for GridStatus to define consistent boundaries across ISOs, however arbitrary they may be. But the raw data should also be available to enable alternative interpretations.
  • [ ] Related to the above (and probably subordinate to it), in ERCOT, GridStatus defines "active" and "completed" using queue["IA Signed"].isna(). While in line with some ISO's taxonomies, I think that is actually inconsistent with ERCOT's definition. ERCOT has a separate sheet for "inactive" projects (not processed by GridStatus), from which I infer that the projects processed by GridStatus are "active".

TrentonBush avatar Nov 08 '23 07:11 TrentonBush

I just opened a PR to address the first two issues here.

bendnorman avatar Nov 15 '23 02:11 bendnorman

thank you for opening the issue @TrentonBush and the detailed notes. I am very open to whatever improvements you want to make. thank you for the PR @bendnorman. i will take a look at over the next couple days

kmax12 avatar Nov 16 '23 18:11 kmax12