API usage with datetimes that aren't at midnight or a multiple of the sensor resolution afterwards

[Flix6x]

Currently, posted data is checked for its resolution, but not for whether events start exactly an integer number of resolutions past midnight (for example, 0, 15, 30 or 45 minutes past the hour, assuming a 15 min resolution). Data sent for a timeslot of 3 past midnight to 18 past midnight is accepted without problems for a sensor with a 15-minute resolution, but doesn't conform to the sensor period in the way that data is handled downstream. We should deal with this.

One straightforward way forward is to disallow such data input, by having the API return an appropriate error message that lays the (arguably small) burden of resampling with zero offset on the API user.

Less straightforward alternatives are:

1. to allow such data for a sensor configured to measure events with a non-zero offset.

In case the sensor would both have some data with and without offset, this would lead to data gaps that are unresolvable, because the gaps have a duration less than the sensor resolution. However, if all data for a given sensor has the same (non-zero) offset, this could still be considered a valid time series with a fixed resolution and could be handled downstream (although requiring additional inference logic).

2. to adjust the sensor to a more fine-grained resolution and to (asynchronously) resample all existing data.

In the example above, 3 minutes would be the greatest common denominator between the offset of 3 minutes and the original resolution of 15 minutes.

[joriszandbergen]

For your information, when triggering a schedule for a battery with the start time or duration not matching the sensor resolution (in my case, 5 minutes), a GET request to the /api/v3_0/sensors/(id)/schedules/(uuid) endpoint either gave back a response with "values": [0, 0, 0, ...] or a 400 error.

[Flix6x]

Thanks for reporting that you bumped into this, hopefully it helps anyone encountering the same issue. Indeed it is not just a problem for posting meter data, but also for triggering schedules within a given time window. The missing validator (for the alignment of the start and duration fields with the sensor's resolution) makes it hard as an API user to find out what went wrong. So we really need an extra validator here.

And to comment on my earlier thought about adjusting the sensor resolution dynamically: that would not be a good solution for datetimes such as start="2022-11-17T15:19:35Z", as one would end up resampling all data to a sensor resolution of 1 second. As long as FlexMeasures is designed around the concept of flow sensors with a fixed resolution, we should resample data to fit the sensor resolution whenever possible, and disallow incompatible time windows.

[Flix6x]

Our new idea is to handle this server-side, at least for triggering schedules, by ceiling the start of the schedule in the . Here's relevant code from v2g-liberty.util_functions.py, and adding it to the start of the endpoint logic in api.v3_0.sensors.trigger_schedule (with a todo to move this logic into a proper schema for this endpoint later).

from datetime import datetime

def time_mod(time, delta, epoch=None):
    """From https://stackoverflow.com/a/57877961/13775459"""
    if epoch is None:
        epoch = datetime(1970, 1, 1, tzinfo=time.tzinfo)
    return (time - epoch) % delta

def time_ceil(time, delta, epoch=None):
    """From https://stackoverflow.com/a/57877961/13775459"""
    mod = time_mod(time, delta, epoch)
    if mod:
        return time + (delta - mod)
    return time

[Flix6x]

Meanwhile, here are some notes to formulate this problem client-side. It sets up the right flex-model to describe the problem in a 15-minute scheduling resolution. The example below assumes a charging capacity of 1% SoC per 15 minutes (and no discharging), so:

• 0.04% in 6 seconds
• 0.8% in 12 minutes