pcmdi_metrics icon indicating copy to clipboard operation
pcmdi_metrics copied to clipboard

Published 20 hours ago • verified publisher icon PCMDI

MJO xcdat conversion

Open lee1043 opened this issue 9 months ago • 5 comments

This PR will resolve #1084.

Draft PR made to track changes. PR is not ready to merge until it is explicitly marked as ready.

lee1043 avatar May 01 '24 23:05 lee1043

Progress note:

The new code was able to run the demo notebook without noticeable error, but with differences in result metrics.

A rigor test (run the code for all CMIP5/6 models/realizations) conducted.

  1. Calendar-related error for non-standard calendar using models is found.
  2. Some differences in the results are found. (Some sensitivity is not surprising because numbers used for the ratio (EWR) are very small.)

Screenshot 2024-05-08 at 6 01 23 PM

Code revision:

  1. To avoid date comparison error from non-standard calendar using models, more simple and straightforward date comparison is applied. It is confirmed the error from the non-standard calendar models is resolved.
  2. Common grid is changed from uniform grid to Gaussian grid to make the xCDAT code more consistent to the CDAT/CDMS version.

Next step:

  • Another rigor test (run the code for all CMIP5/6 models/realizations) is launched. Its result will be analyzed once the test run is completed.
  • ~~Custom season capability when crossing calendar year is not been fully tested, and there to be potential issue (maybe using discontinued months, which could affect to NDJFMA season)~~ → There should be no issue with custom season because for MJO it sub-slice segment using length of segment from the starting day of each year. (here)

lee1043 avatar May 09 '24 01:05 lee1043

  • Usage of more consistent grid (uniform --> Gaussian) did not dramatically changed results (Compare below with above, which are for first ensemble members of CMIP5 (upper row) and 6 (lower row) models).

  • CDAT vs xCDAT results seem to be acceptable (with high correlation and p<0.05), but more investigation for a few outliers would be helpful because their diagnostics fields are looking like identical (see second below figure).

Screenshot 2024-05-14 at 1 00 01 AM Screenshot 2024-05-14 at 1 00 15 AM

lee1043 avatar May 14 '24 08:05 lee1043

  • Progress note: after https://github.com/PCMDI/pcmdi_metrics/pull/1091/commits/f45530781b579664c3e98c3df9171213b71e9233, result of demo notebook 5 became much more consistent than before.
  • The 3rd round of the rigorous test (running across all CMIP5/6 models) is in progress.

lee1043 avatar May 14 '24 21:05 lee1043

The above fix completed resolved the inconsistency! The 3rd round of he rigorous test results below.

  • East West Power Ratio:

Screenshot 2024-05-14 at 10 10 22 PM

  • East Power:

Screenshot 2024-05-14 at 10 10 31 PM

  • West Power:

Screenshot 2024-05-14 at 10 10 40 PM

lee1043 avatar May 15 '24 05:05 lee1043

@acordonez I believe this PR is ready. Can you please try install code in this PR and run the demo 5 notebook to see if that works okay to you as well?

lee1043 avatar May 15 '24 05:05 lee1043

@lee1043 Sorry for taking a few days to get to this! I ran the notebook and found no issues there. I remember you said it was expected to see some slight differences in the metrics? E.g. an east power of 0.016568144258499968 versus 0.016600374830362644?

acordonez avatar Jun 06 '24 18:06 acordonez

@lee1043 Sorry for taking a few days to get to this! I ran the notebook and found no issues there. I remember you said it was expected to see some slight differences in the metrics? E.g. an east power of 0.016568144258499968 versus 0.016600374830362644?

@acordonez thank you for reviewing. Yes, I consider that the difference you found is subtle enough to be neglected.

lee1043 avatar Jun 06 '24 19:06 lee1043