flexmeasures icon indicating copy to clipboard operation
flexmeasures copied to clipboard

Derive data normalization from sensor attributes

Open Flix6x opened this issue 2 years ago • 0 comments

Current status:

  • Our model_spec_factory uses a BoxCoxTransformation on sensor data that is non-negative, and often zero, like wind speed and irradiance.
  • The decision to apply this transformation is currently based on the asset type or sensor name.
  • Migrated power sensors may have attributes (stored in the attributes column) is_producer and is_consumer.
  • All sensors have properties is_strictly_non_positive and is_strictly_non_negative, which are derived from is_producer and is_consumer (if available, and False otherwise).
  • In practice, sensor data may have calibration errors leading to negative values being reported for non-negative sensors.
  • Sometimes producers report consumption values, too (e.g. a wind turbine in maintenance mode).

From https://github.com/SeitaBV/flexmeasures-openweathermap/pull/1#discussion_r807876445 it's clear that we want to base the decision (of what transformation to apply) on more explicit sensor attributes. My proposal is to not store the transformation as a new sensor attribute, but rather to derive the transformation from sensor properties like is_strictly_non_positive and is_strictly_non_negative. We can expand the logic in those 2 properties to also look at whether the sensor attributes contains those exact keys explicitly.

Flix6x avatar Feb 23 '22 08:02 Flix6x