openclimatefix
Add support for evaluating on ExtremeWeatherBench
We want to know how well the models work on extreme events, as well as averages. https://github.com/brightbandtech/ExtremeWeatherBench is a new benchmark for it that would be helpful.
Ok thanks. Just for clarification, is this just a case of creating a new Dataloader that can load in the data in the EWB data format, and then making sure the model can generate predictions in the correct format?
Ok thanks. Also, would you be able to assign me to the issue so I can create a branch and start working on this please?
You should be able to fork this repo, make the changes, and then open a PR back with the updates. That would be the preferred way for this.
@jacobbieker, apologies for the delay, I have only recently managed to start working on this. One thing I have noticed is that when I run the example ExtremeWeatherBench script from the README I get an error no matter which example forecast I use. Do you know for sure that the ExtremeWeatherBench evaluation code works properly? I have also had some trouble working out how to load the data in the correct format as there isn't much documentation available and this is also my first time dealing with numerical weather prediction data.
@sam2206hay Sorry for the delay, I missed this. I'm not entirely sure the evaluation code works properly, but I think it does.
@SkaarFacee this hasn't been finished yet, so feel free to work on it if you want!
@jacobbieker Just to make sure, did @sam2206hay start this and not finish it ? If the work is half way done can I continue from there or would you suggest to start from the beginning ?
Hey, I'm not sure the status of what @sam2206hay has done. I would probably suggest start from the beginning.
I think @SkaarFacee is working on this, but if not, go right ahead! I'd probably say though, if you are thinking of this one and #102, I would start on #102 first, since @SkaarFacee might be working on this one
@jacobbieker understood I would focus on #102 and/or other issues for now!
@jacobbieker @Wendoom-dev Unfortunately I haven't started work on this. I am caught up with another project. If you have the time you may proceed with this
@jacobbieker,
I’m interested in contributing and wanted your advice — would you suggest I start with the ExtremeWeatherBench issue (#133) or continue focusing on the Anemoi Datasets issue (#102), which I’ve already committed to?
Thanks!
@jacobbieker understood I will get started working on #102, thanks for your patience and support!
@jacobbieker I finished #102 should I start work on this or can you suggest any other beginner friendly issue I can contribute to?
Thanks @jacobbieker I would be happy to take this up! Just for my understanding, I need to add support for a system which essentially runs extreme tests on our models? Also are there any previous PRs on this issue that I should refer to or start from scratch?
If you look at the ExtremeWeatherBench repo, the issue is to get it so we can run the models in this repo on ExtremeWeatherBench. It would be starting from scratch
Hlo @jacobbieker, I'm facing the same issue described here: https://github.com/brightbandtech/ExtremeWeatherBench/issues/106 I've encountered this issue on both Windows and Mac M1. I followed the README instructions, but the evaluation still fails. Any suggestions would be helpful.
Hmm, okay, might just need to wait for their response, I'd expect that they would get back to you pretty quick on it, but I'm not sure how to fix that issue unfortunately.
Hmm, okay, might just need to wait for their response, I'd expect that they would get back to you pretty quick on it, but I'm not sure how to fix that issue unfortunately.
Ok, I have some solns in mind to fix the issue, but I am just waiting for their response.
Thanks for the catch. If you can hold off for a little bit there is an incoming refactor and that should address both issues alongside QoL improvements!
@jacobbieker, I think I have to wait as they are working on it. Issue Link: #106