openaq-fetch icon indicating copy to clipboard operation
openaq-fetch copied to clipboard

Update to Rwanda REMA data source

Open majesticio opened this issue 11 months ago • 5 comments

Updates to our Rwanda source, uses a new data URL

majesticio avatar Mar 08 '24 21:03 majesticio

A few things to do before we can push this one

  1. I updated the way that we are specifying the units. Moving forward lets do it this way for all new/updated adapters.
  2. Is there a way to get the siteid in there as the source_id? For example
"title": "Rusizi", "site_id": "RSZ"
  1. Is there a reason that you are filtering out the zeros from the measurements?
  2. Update the documentation for this adapter based on our new template. I would like to see more details about the units and why we have chosen the units we have. If there is data already in the database then lets look at the range of that data and include it.

I added a method that will summarize the parameters while doing the dry run, the output will look something like this

  {
    "message": "[Dry Run] New measurements found for rwanda-rema: 890",
    "failures": {},
    "count": 890,
    "duration": 0.117,
    "from": "2024-03-14T23:39:15.000Z",
    "to": "2024-03-14T23:39:15.000Z",
    "parameters": {
      "co": {
        "min": 0,
        "max": 766.23,
        "nulls": 0,
        "errors": 0,
        "count": 150
      },
      "no2": {
        "min": 0,
        "max": 38.01,
        "nulls": 0,
        "errors": 0,
        "count": 150
      },
      "o3": {
        "min": 0,
        "max": 57.47,
        "nulls": 0,
        "errors": 0,
        "count": 140
      },
      "pm10": {
        "min": 0,
        "max": 71.46,
        "nulls": 0,
        "errors": 0,
        "count": 150
      },
      "pm25": {
        "min": 0,
        "max": 60.71,
        "nulls": 0,
        "errors": 0,
        "count": 150
      },
      "so2": {
        "min": -218.94,
        "max": 59.97,
        "nulls": 0,
        "errors": 0,
        "count": 150
      }
    },
    "sourceName": "rwanda-rema"
  }
]

That should help with figuring out the units.

caparker avatar Mar 14 '24 21:03 caparker

That all sounds good. I was seeing a lot of zero values being returned which I interpreted to mean that there was no data available, which was assumption on my part. I will stop filtering for 0 values.

majesticio avatar Mar 14 '24 22:03 majesticio

Requested changes have been made and merge conflicts resolved

majesticio avatar Mar 20 '24 22:03 majesticio

Added a try/catch block as requested

majesticio avatar Mar 22 '24 22:03 majesticio

There is an issue with time stamps being ahead with this adapter. The provided timestamps eg 2024-04-16T17:32:45Z imply that this is UTC time. However, the resulting measurements are slightly ahead of current time. Additionally, looking at the timestamps ignoring UTC timezone, they align completely with the local time in Rwanda.

I have updated the code to compensate for this discrepancy and will add a note in the documentation

majesticio avatar Apr 16 '24 16:04 majesticio