pudl icon indicating copy to clipboard operation
pudl copied to clipboard

core_eia860m__changelog_generators contains NULL valid_until_date values

Open TrentonBush opened this issue 2 months ago • 4 comments

Describe the bug

core_eia860m__changelog_generators contains 950+ NULL values in the valid_until_date column. I don't think they should exist because this column is derived from report_date and none of those are NULL. Most of the NULLs (700+) are from the last report_date in the series (currently 2023-12-01), but ~200 others exist throughout the time span of the data. Maybe it has to do with the last date in each generator's series?

Bug Severity

This table is encoded as a changelog between two dates -- valid_until_date specifies one of the boundaries. It is hard to interpret the data without that boundary.

To Reproduce

SELECT
    valid_until_date,
    count(*) as count
FROM core_eia860m__changelog_generators
GROUP BY 1
ORDER BY 1 desc

Software Environment?

in pudl.sqlite version v2024.2.6

TrentonBush avatar May 06 '24 17:05 TrentonBush