flexmeasures
flexmeasures copied to clipboard
Derive data normalization from sensor attributes
Current status:
- Our
model_spec_factory
uses aBoxCoxTransformation
on sensor data that is non-negative, and often zero, like wind speed and irradiance. - The decision to apply this transformation is currently based on the asset type or sensor name.
- Migrated power sensors may have attributes (stored in the attributes column)
is_producer
andis_consumer
. - All sensors have properties
is_strictly_non_positive
andis_strictly_non_negative
, which are derived fromis_producer
andis_consumer
(if available, and False otherwise). - In practice, sensor data may have calibration errors leading to negative values being reported for non-negative sensors.
- Sometimes producers report consumption values, too (e.g. a wind turbine in maintenance mode).
From https://github.com/SeitaBV/flexmeasures-openweathermap/pull/1#discussion_r807876445 it's clear that we want to base the decision (of what transformation to apply) on more explicit sensor attributes. My proposal is to not store the transformation as a new sensor attribute, but rather to derive the transformation from sensor properties like is_strictly_non_positive
and is_strictly_non_negative
. We can expand the logic in those 2 properties to also look at whether the sensor attributes contains those exact keys explicitly.