uxarray icon indicating copy to clipboard operation
uxarray copied to clipboard

E3SM UXarray and Dask Examples

Open rytam2 opened this issue 1 year ago • 55 comments
trafficstars

Closes #822

Overview

Added user guide notebook on parallel loading with Dask; and workflow examples with E3SM output with visualization and calculations.

PR Checklist

General

  • [x] An issue is linked created and linked
  • [x] Add appropriate labels
  • [x] Filled out Overview and Expected Usage (if applicable) sections

Examples

  • [x] Any new notebook examples added to docs/examples/ folder
  • [x] Clear the output of all cells before committing
  • [x] New notebook files added to docs/examples.rst toctree
  • [x] New notebook files added to new entry in docs/gallery.yml with appropriate thumbnail photo in docs/_static/thumbnails/

rytam2 avatar Jul 03 '24 18:07 rytam2

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-08T16:41:20Z ----------------------------------------------------------------

Maybe we could change the title to something that better reflects the content of the notebook?

"Analysis of E3SMv2 Model Output"


View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-08T16:41:21Z ----------------------------------------------------------------

Overview

This workflow example showcases how to use UXarray to analyze the unstructured grid output from the Energy Exascale Earth System Model (E3SM) model directly without needing to perform any regridding operations.


View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-08T16:41:22Z ----------------------------------------------------------------

Line #22.    warnings.filterwarnings("ignore")

We want to have this warning filter be the first thing we execute so that none of the warnings show up.


View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-08T16:41:23Z ----------------------------------------------------------------

This is great!


View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-08T16:41:24Z ----------------------------------------------------------------

The latex appears to be broken in the rendered documentation.

https://uxarray--836.org.readthedocs.build/en/836/examples/e3sm-calc-workflow.html#calculating-cloud-radiative-effect-netcre-with-uxarray

You want to use the following syntax instead (for example)

{math}4{\pi}{r^2}


rytam2 commented on 2024-07-08T22:16:31Z ----------------------------------------------------------------

Hi Philip, thanks for the comment! For the syntax, do you mean I'll do the following:

$$
{SWCRE = FSNT-FSNTC}
$$

Or do you mean adding {math} before the SWCRE...FSNTC equation, and have $$ removed?

philipc2 commented on 2024-07-09T06:22:00Z ----------------------------------------------------------------

Correct, it should look like {math}{SWCRE = FSNT - FSNTC}

View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-08T16:41:25Z ----------------------------------------------------------------

I do like the idea of having this example of plots in a single row compared to the stacked ones below, but I think for consistency we should keep these plots the same as the ones below.


View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-08T16:41:25Z ----------------------------------------------------------------

Line #45.        * features.opts(fig_size=160)

This applies to the other plots too. Can we try a slightly larger fig_size.Maybe 250?


Hi Philip, thanks for the comment! For the syntax, do you mean I'll do the following:

$$
{SWCRE = FSNT-FSNTC}
$$

Or do you mean adding {math} before the SWCRE...FSNTC equation, and have $$ removed?


View entire conversation on ReviewNB

rytam2 avatar Jul 08 '24 22:07 rytam2

Correct, it should look like {math}{SWCRE = FSNT - FSNTC}


View entire conversation on ReviewNB

philipc2 avatar Jul 09 '24 06:07 philipc2

Looks like the LaTex is still broken. Can you add the following to our conf.py and see if it fixes things?

https://github.com/UXARRAY/uxarray/blob/main/docs/conf.py

myst_enable_extensions = [
    "amsmath",
    "attrs_inline",
    "colon_fence",
    "deflist",
    "dollarmath",
    "fieldlist",
    "html_admonition",
    "html_image",
    "linkify",
    "replacements",
    "smartquotes",
    "strikethrough",
    "substitution",
    "tasklist",
]

philipc2 avatar Jul 11 '24 21:07 philipc2

@rytam2

Now that you've marked the PR as ready for review, there's a few things that you can also do:

  • Remove the DRAFT from the title
  • Add reviewers (can add orhan, rajeev, aaron and myself)
image

philipc2 avatar Jul 12 '24 16:07 philipc2

@anissa111

Any idea on why LatEx isn't rendering correctly in the notebooks? We've tried a few things above but are still having some issues.

philipc2 avatar Jul 12 '24 17:07 philipc2

@anissa111

Any idea on why LatEx isn't rendering correctly in the notebooks? We've tried a few things above but are still having some issues.

I'll take a look!

anissa111 avatar Jul 12 '24 17:07 anissa111

Okay, locally either

```{math}
	SWCRE = FSNT - FSNTC
```

or

$$
	SWCRE = FSNT - FSNTC
$$

works for me

anissa111 avatar Jul 12 '24 18:07 anissa111

View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-17T00:18:36Z ----------------------------------------------------------------

... as follows:


View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-17T00:18:36Z ----------------------------------------------------------------

Line #1.    # Calculate Shortwave Cloud Radiative Effect (SWCRE) - lazily loaded

Can you turn these comments into a heading? Maybe with the "###" size?


View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-17T00:18:37Z ----------------------------------------------------------------

The figure size is still pretty small for these. Can we try something around 200 to 250?


View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-17T00:18:38Z ----------------------------------------------------------------

Line #1.    # Calculate Longwave Cloud Radiative Effect (LWCRE)

Same as comment above.


View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-17T00:18:38Z ----------------------------------------------------------------

Line #1.    # Calculate Net CRE

Same as comment above


View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-17T00:18:40Z ----------------------------------------------------------------

For user guide sections, we don't need this title / description.


View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-17T00:18:40Z ----------------------------------------------------------------

This applies throughout the notebooks. I'd suggest moving any comments that you have in the code blocks into the markdown above, unless otherwise necessary.


View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-17T00:18:41Z ----------------------------------------------------------------

Any reason to why you only used a client here with no cluster? I'd also suggest removing any commented out lines of code from the final notebook.


rytam2 commented on 2024-07-17T21:14:54Z ----------------------------------------------------------------

Now updated with cluster usage.

View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-17T00:18:42Z ----------------------------------------------------------------

This seems to be in the wrong place?


rytam2 commented on 2024-07-17T21:15:36Z ----------------------------------------------------------------

Moved above prior to closing cluster.

Now updated with cluster usage.


View entire conversation on ReviewNB

rytam2 avatar Jul 17 '24 21:07 rytam2

Moved above prior to closing cluster.


View entire conversation on ReviewNB

rytam2 avatar Jul 17 '24 21:07 rytam2

View / edit / reply to this conversation on ReviewNB

erogluorhan commented on 2024-07-22T20:10:16Z ----------------------------------------------------------------

"Energy Exascale Earth System Model (E3SM) model" -> remove the second "model"


View / edit / reply to this conversation on ReviewNB

erogluorhan commented on 2024-07-22T20:10:16Z ----------------------------------------------------------------

That'd be nice to point the reader out to the Dask user guide before Dask setup starts here.


View / edit / reply to this conversation on ReviewNB

erogluorhan commented on 2024-07-22T20:10:18Z ----------------------------------------------------------------

"... how to load unstructured input data in parallel with ..."


View / edit / reply to this conversation on ReviewNB

erogluorhan commented on 2024-07-22T20:10:18Z ----------------------------------------------------------------

Why and in what use cases parallel would make a difference is not discussed anywhere in this notebook. I think we should share a takeaway about it somewhere for the reader


rytam2 commented on 2024-07-26T04:22:11Z ----------------------------------------------------------------

Added paragraph from zulip discussion.