🐛[BUG]: GraphCast Model Registry Issues

[david5010]

Version

main

On which installation method(s) does this occur?

Source

Describe the issue

Currently, it looks like GraphCast metadata can't be downloaded correctly. It currently fails when running the graphcast_simple.py but it looks like it downloaded some parameters and stats needed for it to run. My guess is that the download and caching logic might be faulty for GraphCast. Here's what the parameters and stats look like in my .cache/

If anyone else has solved this issue, please let me know how you ran graphcast!

Environment details

pip install .
pip install .[graphcast]
pip install -r requirements.txt

[nbren12]

Does it still fail if you clean up the cache. I recently use graphcast, but manually downloaded it with gsutil into a $MODEL_REGISTRY. That workflow may be more robust.

[david5010]

It still fails. I've tried to download the weights and stats directly from Google's repository but I don't know how to format it correctly for this use case. Would you be able to share how you manually downloaded with gsutil @nbren12 ? I also can't find gsutil in the repository

[nbren12]

Gsutil is google’s tool for working with their cloud storage. You should be able to find install and usage instructions with a search.

IIRC you need something like this

MODEL_REGISTRY=/a/path/of/your/choice/ gsutil cp gs://dm_graphcast/ $MODEL_REGISTRY/graphcast

After this command the directory should look something like

$ find . ./stats ./stats/mean_by_level.nc ./stats/stddev_by_level.nc ./stats/diffs_stddev_by_level.nc ./params ./params/GraphCast - ERA5 1979-2017 - resolution 0.25 - pressure levels 37 - mesh 2to6 - precipitation input and output.npz ./metadata.json ./dataset ./dataset/source-era5_date-2022-01-01_res-0.25_levels-37_steps-04.nc

In this example, I only downloaded the params/ file I actually needed.

Then, you need to add a metadata.json file in the graphcast/ folder to tell earth2mip how to open this: { "entrypoint": {"name": "earth2mip.networks.graphcast:load_time_loop"} }

Then, you should be able to use networks.get_model(“graphcast”) or other scripts that take in the model name. Be sure to export the MODEL_REGISTRY environment variable in your shell configuration (.bashrc) so that the setting will persist.

Hopefully this helps. This is a workaround, and I may not be able to help much more than this since I have limited time to help users with earth2mip.

Best, Noah

From: david5010 @.> Date: Tuesday, October 8, 2024 at 6:55 AM To: NVIDIA/earth2mip @.> Cc: Noah Brenowitz @.>, Mention @.> Subject: Re: [NVIDIA/earth2mip] 🐛[BUG]: GraphCast Model Registry Issues (Issue #195)

It still fails. I've tried to download the weights and stats directly from Google's repository but I don't know how to format it correctly for this use case. Would you be able to share how you manually downloaded with gsutil @nbren12https://github.com/nbren12 ? I also can't find gsutil in the repository

— Reply to this email directly, view it on GitHubhttps://github.com/NVIDIA/earth2mip/issues/195#issuecomment-2399918487, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAKSRETAW6AILON2PVXN6HLZ2PP2VAVCNFSM6AAAAABLE6LXZGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJZHEYTQNBYG4. You are receiving this because you were mentioned.Message ID: @.***>