open-grid-emissions
open-grid-emissions copied to clipboard
Fix incomplete timeseries warnings
When running the data pipeline for 2021, there are several warnings appearing for various data outputs regarding incomplete timeseries data:
Data quality metric export:
2023-06-09 20:20:56,968 [INFO] oge.output_data:158 Exporting hourly_profile_method to data/results/2021/data_quality_metrics
2023-06-09 20:20:58,949 [WARNING] oge.validation:792 There are incomplete timeseries for the following ['plant_id_eia'] groups in shaped_eia_data
2023-06-09 20:20:58,949 [WARNING] oge.validation:795
count min max expected_num_hours
plant_id_eia report_date
903212 2021-11-01 719 2021-11-01 08:00:00+00:00 2021-12-01 08:00:00+00:00 721
909312 2021-11-01 719 2021-11-01 05:00:00+00:00 2021-12-01 05:00:00+00:00 721
913612 2021-11-01 719 2021-11-01 03:00:00+00:00 2021-12-01 03:00:00+00:00 721
[...]
Upon combining hourly plant data:
2023-06-09 20:21:59,875 [INFO] oge.data_pipeline:542 16. Combining plant-level hourly data
2023-06-09 20:24:04,908 [WARNING] oge.validation:773 There are incomplete timeseries for the following ['plant_id_eia'] groups in combined_plant_data
2023-06-09 20:24:04,970 [WARNING] oge.validation:776
datetime_utc expected_num_hours
plant_id_eia
9 5136 8760
47 6553 8760
51 6529 8760
59 7296 8760
124 5808 8760
170 5064 8760
201 8017 8760
203 6552 8760
[...]
For certain BA-level results:
2023-06-09 20:24:22,416 [INFO] oge.output_data:131 Exporting AECI to data/results/2021/power_sector_data/annual/
2023-06-09 20:24:22,448 [INFO] oge.validation:149 Checking that fuel and emissions values are positive...
2023-06-09 20:24:22,463 [INFO] oge.validation:175 OK
2023-06-09 20:24:22,463 [INFO] oge.validation:181 Checking that no values are missing...
2023-06-09 20:24:22,463 [INFO] oge.validation:198 OK
2023-06-09 20:24:22,683 [WARNING] oge.validation:773 There are incomplete timeseries for the following ['fuel_category'] groups in power sector hourly ba table
2023-06-09 20:24:22,683 [WARNING] oge.validation:776
datetime_utc expected_num_hours
fuel_category
wind 8758 8760
2023-06-09 20:26:00,361 [INFO] oge.output_data:131 Exporting LDWP to data/results/2021/power_sector_data/annual/
2023-06-09 20:26:00,392 [INFO] oge.validation:149 Checking that fuel and emissions values are positive...
2023-06-09 20:26:00,408 [INFO] oge.validation:175 OK
2023-06-09 20:26:00,408 [INFO] oge.validation:181 Checking that no values are missing...
2023-06-09 20:26:00,408 [INFO] oge.validation:198 OK
2023-06-09 20:26:00,612 [WARNING] oge.validation:773 There are incomplete timeseries for the following ['fuel_category'] groups in power sector hourly ba table
2023-06-09 20:26:00,612 [WARNING] oge.validation:776
datetime_utc expected_num_hours
fuel_category
natural_gas 8761 8760
total 8761 8760