metno
Revise FAIRMODE plots
The FAIRMODE plots should be based on hourly mean data for NO2 (and not on daily or monthly means as they are now), and for ozone they should be based on the 8h moving average daily maximum.
Related issue: #863
@charlienegri is trying to run the cams2_83 collocation with hourly statistics
mda8 is implemented (@charlienegri )
no2 hourly currently takes too long time due to a performance bug in regional statistics #1277 blocking the implementation
@charlienegri made a new test of fairmode / hourly data:
- reading observations is no problem
- current production runs about 12h, all 6-7 variables (O3/O3MDA), 16.6G memory-consumption, only daily
cams283_analysis_long_1_JJA2024.log
- statistics comparinga against one model took 320s, equal share of about 1min for each of the individual statistical tasks
- data-output (json files) is much larger, e.g. ts 15G, hm/ts 3.5G (only no2-files)
- test-run, only no2 but hourly runs about 12.5h, 17.1G max memory consumption
cams283_test_forecast_long_no2only_hourly.log
- statistics comparing against a single model took ~5000s (most in Processing statistics timeseries for all regions 50min but also 20min in Processing individual site timeseries data)
- data-output (json files) is much smaller, e.g. ts 0.8G, hm/ts 0.2G (only no2-files)
Speed is partly connected to the output-size, so just disabling the hourly timeseries (only daily) might speed things up a lot. @michaelgau We should sit together and discuss, which output needs to be available hourly. For (most) all others plots, we should stick to daily. Also plotting hourly data might get very slow of so huge datasets.
ok, thanks! Will look at this when I'm back from the CAMS2_61 meeting (14 Nov)
After discussion with @michaelgau
For the "last seasons" plot, hourly output is in general not needed. Only the Fairmode NO2 plots should be based on hourly data. In particular: Fairmode plots are only defined for one time resolution:
- daily:
- O3-MDA
- PM10
- PM2.5
- hourly:
- NO2
e.g. monthly PM10 plots are wrong because they are based on threshold values which are only valid for daily data etc. This means, a time-resolution selector for Target-plots does not make sense, all undefined plots should be removed (as they are already for CO/SO2).
yeah Michael has explained us this before, my understanding was that it was up to the users to select the frequency that makes sense, if available. if meaningless selections can be hidden front-end side, great.
about NO2, so far there is no way of running hourly for fairmode statistics only
just for the record: FAIRMODE plots do exist also for annual time resolutions, but these are not relevant for us because we don't need to make annual evaluations on CAMS2_83 aeroval.