TomboloDigitalConnector
TomboloDigitalConnector copied to clipboard
ConnectedPlacesCatapult
Multiple values with same timestamp are not getting saved
Description
If there are multiple values with same timestamp, only the first one is getting saved and rest all are getting ignored. e.g if you get london air quality data on hourly basis, it takes multiple reading within the hour and api returns all the multiple ratings but when DC tries to save it, it only picks the first one and ignores the rest all.
Error log
No error
I am not sure if this is a bug. There should only be one value per timestamp.
As for the LAQN, each reading should have the actual timestamp when the reading was made. That should be the timestamp of the data, not the timestamp when the api endpoint was called.
Archive.zip @borkurdotnet that should ideally be the case, but london air quality also provides datasets with hourly and daily values. Datasets that provide values for whole day, only contains date and multiple readings but not the timestamp at which the reading was taken. Datasets that provides values on hourly basis, provides one timestamp for that hour and if there are multiple reading taken during that hour, timestamp doesn't change. e.g if there are 3 readings taken between 10-11am then the timestamp only shows 10am and multiple entries of values. Both type of datasets are attached.
In that case this could be done by using different attributes. The hourly and daily values are two different time series that should be two separate time series. The daily time-series could have timestamp at noon or midnight.
Regarding the multiple reading per timestamp in hourly values, I would contact the LAQN and ask what those values mean since they appear only for two sites: WM5 and WM6. If they are indeed multiple readings during the hour, I'd look into aggregating them into a single hourly value rather than extending the centralised data format for this corner case.
Point taken, i will check with KCL tomorrow, however post check there other api calls looks like they just have more sensors installed in that area, but i will confirm with them. But if the more values is because of more sensors then i believe this could happen with other areas as well. This is one corner case we have found, there could be more as people could build importers for datasets we may not know yet. It would be great if we give them as much flexibility as possible.
@borkurdotnet @arya-hemanshu I see this scenario falling in the real-time support (e.g. #131 and similar in the Real Time API support milestone).
