openaq-fetch
openaq-fetch copied to clipboard
Update to Rwanda REMA data source
Updates to our Rwanda source, uses a new data URL
A few things to do before we can push this one
- I updated the way that we are specifying the units. Moving forward lets do it this way for all new/updated adapters.
- Is there a way to get the siteid in there as the source_id? For example
"title": "Rusizi", "site_id": "RSZ"
- Is there a reason that you are filtering out the zeros from the measurements?
- Update the documentation for this adapter based on our new template. I would like to see more details about the units and why we have chosen the units we have. If there is data already in the database then lets look at the range of that data and include it.
I added a method that will summarize the parameters while doing the dry run, the output will look something like this
{
"message": "[Dry Run] New measurements found for rwanda-rema: 890",
"failures": {},
"count": 890,
"duration": 0.117,
"from": "2024-03-14T23:39:15.000Z",
"to": "2024-03-14T23:39:15.000Z",
"parameters": {
"co": {
"min": 0,
"max": 766.23,
"nulls": 0,
"errors": 0,
"count": 150
},
"no2": {
"min": 0,
"max": 38.01,
"nulls": 0,
"errors": 0,
"count": 150
},
"o3": {
"min": 0,
"max": 57.47,
"nulls": 0,
"errors": 0,
"count": 140
},
"pm10": {
"min": 0,
"max": 71.46,
"nulls": 0,
"errors": 0,
"count": 150
},
"pm25": {
"min": 0,
"max": 60.71,
"nulls": 0,
"errors": 0,
"count": 150
},
"so2": {
"min": -218.94,
"max": 59.97,
"nulls": 0,
"errors": 0,
"count": 150
}
},
"sourceName": "rwanda-rema"
}
]
That should help with figuring out the units.
That all sounds good. I was seeing a lot of zero values being returned which I interpreted to mean that there was no data available, which was assumption on my part. I will stop filtering for 0 values.
Requested changes have been made and merge conflicts resolved
Added a try/catch block as requested
There is an issue with time stamps being ahead with this adapter. The provided timestamps eg 2024-04-16T17:32:45Z
imply that this is UTC time. However, the resulting measurements are slightly ahead of current time. Additionally, looking at the timestamps ignoring UTC timezone, they align completely with the local time in Rwanda.
I have updated the code to compensate for this discrepancy and will add a note in the documentation