openaq-fetch
openaq-fetch copied to clipboard
openaq
malfunctioning adapters
Adapters that need attention
This is a list of adapter names that are not returning data properly with their known errors. Does not include ones set as 'active : false' @caparker
- au
- Australia - South Australia
- unknown adapter error
- Australia - Victoria: Failure to load data url
- Australia - South Australia
- ba
- Bosnia_Tuzlanski <- easy fix?
- 4 occurrences of instance requires property "unit"
- 4 occurrences of instance.parameter is not one of enum values: pm25,pm10,no2,so2,o3,co,bc
- Bosnia_Tuzlanski <- easy fix?
- be
- EEA Belgium? https://discomap.eea.europa.eu/map/fme/latest/ works
- bg
- EEA Bulgaria https://discomap.eea.europa.eu/map/fme/latest/ same as BE
- br
- Sao Paulo CETESB): Cannot read property 'measurements' of undefined - https://qualar.cetesb.sp.gov.br/qualar/home.do -> needs login?
- ch
- EEA Switzerland - https://discomap.eea.europa.eu/map/fme/latest/ same as BG BE
- co
- Bogota: Failed to load entry point url http://api-aire.ambientebogota.gov.co/api/sda/openAq SERVICE UNAVAILABLE
- de
- EEA Germany - https://discomap.eea.europa.eu/map/fme/latest/ EEA website, again
- dk
- EEA Denmark
- es
- EEA Spain (EU again)
- Andalucia
- http://www.juntadeandalucia.es/medioambiente/site/portalweb/menuitem.7e1cf46ddf59bb227a9ebe205510e1ca/?vgnextoid=7e612e07c3dc4010VgnVCM1000000624e50aRCRD&vgnextchannel=3b43de552afae310VgnVCM2000000624e50aRCRD
- REDIRECTS TO https://www.juntadeandalucia.es/medioambiente/portal/home
- ge
- EEA Georgia
- gi
- EEA Gibraltar
- is
- Umhverfisstofnun
- cert expired
- Umhverfisstofnun
- lt
- EEA Lithuania (EU!!!!)
- rw
- Rwanda REMA
- empty site ""
- Rwanda REMA
Great list. I am checking each of them against the lambda branch and then will make a new issue for each so that they can be fixed and/or closed individually.
Australia - Victoria Working on the lambda branch
node index.js -e .env -ds 'Australia - Victoria'
{"level":"info","message":"--- Dry run for Testing, nothing is saved to the database. ---"}
{"level":"info","message":"Looking up source Australia - Victoria"}
{"level":"info","message":"[Dry Run] File would be saved to realtime/2022-10-27/1666879891.ndjson"}
{"level":"info","message":"New measurements found for \"Australia - Victoria\": 68 in 0.048s"}
{"level":"info","message":{"errors":{},"itemsInserted":68,"results":[{"count":68,"duration":0.048,"failures":{},"message":"[Dry Run] New measurements found for Australia - Victoria: 68","sourceName":"Australia - Victoria"}],"timeEnded":1666879943839,"timeStarted":1666879891091}}
{"level":"info","message":"Dry run ended."}
Though I will say it takes up to a minute to get the response and that is not reflected in that log message above, it is correct in hte timeEnded - timeStarted but that doesnt seem to be reported. I recommend that we create an warning code for long adapters given that we only have 15min to work with in the lambda. The warning would help us identify the sources that we need to pull out to their own lambda.
We could also update this adapters error handling We could also merge this adapter with the other australian adapters because on first glance they seem pretty similar, differing only in config. Could be wrong there though :)
@caparker I have been going through the backlog of issues, it seems that issue #690 may be why there is a lot of data missing from EEA adapters (Germany is mentioned). I will be looking into this but please let me know if you have any thoughts.