staged-recipes
staged-recipes copied to clipboard
pangeo-forge
NOAA GFDL sea ice concentration data assimilation fields
Model outputs and input observations from GFDL's sea ice concentration data assimilation system. These files include the daily model forecasts (70 different state vectors and diagnostic variables), daily satellite observations, the data assimilation prior states, posterior states, and subsequently data assimilation increments (posterior - prior). Data assimilation is performed every 5 days (where observations are available), beginning on January 6th 1982. Leap years are removed from all fields
It looks like your meta.yaml does not conform to the specification.
2 validation errors for MetaYaml
pangeo_notebook_version
field required (type=value_error.missing)
provenance -> providers -> 0 -> url
URL invalid, extra characters found after valid URL: ' (model) / https://nsidc.org/data/NSIDC-0051/versions/1 (obs)' (type=value_error.url.extra; extra= (model) / https://nsidc.org/data/NSIDC-0051/versions/1 (obs))
Please correct your meta.yaml and commit the corrections to this PR.
It looks like your meta.yaml does not conform to the specification.
1 validation error for MetaYaml
pangeo_notebook_version
field required (type=value_error.missing)
Please correct your meta.yaml and commit the corrections to this PR.
It looks like your meta.yaml does not conform to the specification.
1 validation error for MetaYaml
pangeo_notebook_version
field required (type=value_error.missing)
Please correct your meta.yaml and commit the corrections to this PR.
:tada: New recipe runs created for the following recipes at sha 56db201a416fd9cf5323b0895b8cbce75d8f8edf:
sic-da: https://pangeo-forge.org/dashboard/recipe-run/356
@William-gregory, re: https://github.com/pangeo-forge/staged-recipes/pull/139#issuecomment-1154342977: Ultimately, whatever recipes are included in a single PR will end up grouped together in a distinct feedstock repository. If an outside user were to stumble across one of these feedstock repositories, would the grouping of datasets contained therein make sense to them? That is, would the datasets appear to be coherent group, apart from their usefulness to your current research project? If so, let's combine them in a single PR. If not, separate PRs makes sense.
Regarding this particular PR, now that we've got a successful recipe run, the next step is that I can trigger a test of that recipe on Pangeo Forge Cloud. As it happens, over in https://github.com/pangeo-forge/staged-recipes/pull/138 I'm working on some infrastructure updates which seem to have temporarily broken something about Pangeo Forge Cloud's job submission process. Once I have that sorted out, I'll come back here and trigger the test run of sic-da.
hey @cisaacstern, sorry to be a pain, but if we could hold off on the test run for a day or two would be great. At the moment the data which are being called in the recipe.py file are stored in a directory on an ftp server which is routinely wiped, which if I understand right is a problem for pangeo-forge? as any data pulls in the future will always be searching from this ftp location? I've just been granted access to an alternative directory which is not wiped, so I'm just copying the data there now. After it's finished copying I'll update the recipe.py file and make another pull request.
Yes of course no problem
After it's finished copying I'll update the recipe.py file and make another pull request.
You could also make additional commit(s) to this PR, updating the file pattern appropriately, which may be easier than opening a new one. Either way, just ping me when you're ready to test, and as always feel free to reach out with any questions.
And yes, to answer your question, it is preferable for the source data to exist in a persistent location.
hey @cisaacstern , just pinging you here. I changed the file paths in my recipe.py file, but it seems there's a failure on the automated checks. I thought this might be a permissions issue, so I went back to the ftp site and made sure I opened permissions on all directories, but unsure how to proceed from here. Thanks!
@William-gregory - the "bad credentials" error is a problem on our side. We will fix it asap and re-run the check. Thanks for your patience!
Yes apologies for that! I am not at my computer at the moment, but as soon as I am, I'll replace the credentials and re-run the check.
:tada: New recipe runs created for the following recipes at sha b7ff0925abd9d8c05df2f5fc14575d1d0f381668:
sic-da: https://pangeo-forge.org/dashboard/recipe-run/364
Problem resolved. @William-gregory, are you ready to run a test of this recipe now or would you prefer that I wait?
When I tried to import your recipe module, I encountered this error
line 34, in <module>
recipe3 = XarrayZarrRecipe(pattern3, target_chunks={'time':97})
File "<string>", line 21, in __init__
File "/usr/local/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 832, in __post_init__
raise ValueError(
ValueError: Unable to determine target chunks. Please specify either `target_chunks` or `nitems_per_input`
Please correct your recipe module so that it's importable.
:tada: New recipe runs created for the following recipes at sha 84fb0691ca6f356d0be6801913186062f16df246:
sic-da: https://pangeo-forge.org/dashboard/recipe-run/365
:tada: New recipe runs created for the following recipes at sha b6de9dfec1b53e1aa700e69cd0dcdcb31d71e330:
sic-obs: https://pangeo-forge.org/dashboard/recipe-run/366sic-da-forecasts: https://pangeo-forge.org/dashboard/recipe-run/367sic-da-increments: https://pangeo-forge.org/dashboard/recipe-run/368sic-da-priors: https://pangeo-forge.org/dashboard/recipe-run/369sic-da-posteriors: https://pangeo-forge.org/dashboard/recipe-run/370
I'll just start with a test of these two, and then do the other three if/when these succeed.
:sparkles: A test of your recipe sic-da-forecasts is now running on Pangeo Forge Cloud!
I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)
In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/367
:sparkles: A test of your recipe sic-obs is now running on Pangeo Forge Cloud!
I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)
In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/366
Pangeo Forge Cloud told me that our test of your recipe sic-da-forecasts failed. But don't worry, I'm sure we can fix this!
To see what error caused the failure, please review the logs at https://pangeo-forge.org/dashboard/recipe-run/367
If you haven't yet tried pruning and running your recipe locally, I suggest trying that now.
Please report back on the results of your local testing in a new comment below, and a Pangeo Forge maintainer will help you with next steps!
Pangeo Forge Cloud told me that our test of your recipe sic-obs failed. But don't worry, I'm sure we can fix this!
To see what error caused the failure, please review the logs at https://pangeo-forge.org/dashboard/recipe-run/366
If you haven't yet tried pruning and running your recipe locally, I suggest trying that now.
Please report back on the results of your local testing in a new comment below, and a Pangeo Forge maintainer will help you with next steps!
I'm sorry that was my fault 🤦 . I've been working on the backend code, and some changes I'd made there apparently caused things to blow up. This has nothing to do with the recipes themselves. I will fix this and then re-run the tests.
Apologies for the confusion. I believe the problem is now fixed on Pangeo Forge Cloud. Will re-try the same tests again now.
:sparkles: A test of your recipe sic-obs is now running on Pangeo Forge Cloud!
I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)
In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/366