cumulus
cumulus copied to clipboard
nasa
Hyrax update fails when a CMR metadata record doesn't exist in the payload
We have some cases where a raw file would create one or more processed files, the original raw file will not be published to CMR (hence no CMR metadata). In this case a raw file f17_202103v7.gz would generate f17_ssmis_202103v7.nc So the payload passed to Hyrax update would look like
"payload": { "granules": [ { "granuleId": "f17_ssmis_202103v7.nc", "files": [ { "path": "rssmif17m__7", "url_path": "rssmif17m__7", "bucket": "ghrcsit-protected", "name": "f17_ssmis_202103v7.nc", "filename": "s3://ghrcsit-protected/rssmif17m__7/f17_ssmis_202103v7.nc", "size": 1215032, "filepath": "rssmif17m__7/f17_ssmis_202103v7.nc" }, { "path": "rssmif17m__7", "url_path": "rssmif17m__7", "bucket": "ghrcsit-public", "name": "f17_ssmis_202103v7.nc.cmr.xml", "filename": "s3://ghrcsit-public/rssmif17m__7/f17_ssmis_202103v7.nc.cmr.xml", "size": 1652, "filepath": "rssmif17m__7/f17_ssmis_202103v7.nc.cmr.xml" }, { "name": "f17_ssmis_202103v7.nc.dmrpp", "path": "rssmif17m__7", "url_path": "rssmif17m__7", "bucket": "ghrcsit-protected", "size": 59345, "type": "metadata", "filepath": "rssmif17m__7/f17_ssmis_202103v7.nc.dmrpp", "filename": "s3://ghrcsit-protected/rssmif17m__7/f17_ssmis_202103v7.nc.dmrpp" } ] }, { "granuleId": "f17_202103v7.gz", "files": [ { "path": "rssmif17m__7", "url_path": "rssmif17m__7", "bucket": "ghrcsit-private", "name": "f17_202103v7.gz", "filename": "s3://ghrcsit-private/rssmif17m__7/f17_202103v7.gz", "size": 780813, "filepath": "rssmif17m__7/f17_202103v7.gz" } ] }
HuraxUpdate will throw an error No recognizable CMR metadata file (*.cmr.xml or *.cmr.json) for granule f17_202103v7.gz
Can we have HuraxUpdate lmbda just throw a warning instead of failing?
@amarouane-ABDELHAK
Would it be possible to address this issue by skipping the HyraxUpdate step conditionally based on some attribute perhaps of the collection in the message? This seems like the best option if possible, since it seems like you don't really need the Hyrax step for this data?
Or, could you configure the Hyrax step in your Step Function catch this error?
I need hyrax update for this collection as we are producing f17_ssmis_202103v7.nc.dmrpp
I need hyrax update for this collection as we are producing
f17_ssmis_202103v7.nc.dmrpp
You don't need the hyrax update to create the dmrpp file right? All the hyrax task does is augment metadata. If you have no metadata then it could be skipped.
Yes, that is exactly correct. But I don't want to skip the HyraxUpdate step.
But I don't want to skip the HyraxUpdate step. What does this step give you if you have no metadata to update?
But I don't want to skip the HyraxUpdate step. What does this step give you if you have no metadata to update?
Error: No recognizable CMR metadata file (*.cmr.xml or *.cmr.json) for granule.
@amarouane-ABDELHAK
Would it be possible to address this issue by skipping the HyraxUpdate step conditionally based on some attribute perhaps of the collection in the message? This seems like the best option if possible, since it seems like you don't really need the Hyrax step for this data?
Or, could you configure the Hyrax step in your Step Function catch this error?
The configuration of the Step to catch the error and move to post-to-CMR looks promising 🤔
If I can do something like
"Catch": [ { "ErrorEquals": [ "RecordDoesNotExist" ], "Next": "PostToCMR" }, { "ErrorEquals": [ "States.ALL" ], "ResultPath": "$.exception", "Next": "WorkflowFailed" } ]
Using the catch in the step will make the exception propagate to post-to-cmr which will make it fail. I thin it is better to just warn the user there is no CMR metadata (in cloud watch logs) and not to force it to fail.
@amarouane-ABDELHAK Is this still an issue? I'm still not quite understanding what the Hyrax step is doing for you if you don't have any CMR metadata, since all that task is doing is updating CMR metadata with some links for OpenDAP, as far as I know