openaq-fetch
openaq-fetch copied to clipboard
New South Wales, Australia incorrect history
continuing from #646, the NSW, AU data was previously scrapped incorrectly so the data is incorrect, this needs fixing. Quoting from slack https://openaq.slack.com/archives/C098H3JH2/p1576814663003400
...Removing the data and try to do an invest of the older data may be a good way to go, with one caveat. Our system currently has no idea of whether the data is qa/qc’d, but only that it’s real-time. If the older data is not the same ‘class’ as what’s available in real-time, we’d be mixing two different classes of data which may be an issue. Andrew Harvey 1 hour ago ...I'll try to compare the historical data to the live to see if there's a difference.
This is different to #647, which is about loading historical data from even before we started ingesting invalid data.
@russbiggs I am not sure if we can fix the data if it was scraped incorrectly (adapter: "Australia - New South Wales" nsw.js) - unless the source provides historical data. Currently I cannot get the website to resolve https://airquality.environment.nsw.gov.au/ . We should consider erasing the data from this source prior to Dec 19 2019, if not just the entries for so2 from issue #646 (source name: State of New South Wales (Department of Planning, Industry and Environment)
https://airquality.environment.nsw.gov.au/aquisnetnswphp/getPage.php?reportid=2 is working for me