earthaccess

earthaccess copied to clipboard

Hi!

I'm Jorge and im triying to download a few satellite retrievals. When I run the code an error appeared saying "Credentials for the cloud provider GES_DISC are not available". I'm registered in Earth Data Search and I activated the application NASA GESDISC DATA ARCHIVE i dont know if i have to do something else to get that credentials.

this is the code im using:

from earthdata import Auth, DataGranules, Store

auth = Auth().login(strategy='interactive')

granules = DataGranules().concept_id("C1239966842-GES_DISC").temporal("2004-10-01", "2004-10-31").point(-3.64746,37.19949).get_all()

files = Store(auth).get(granules, local_path='./data')

and the code returns:

You're now authenticated with NASA Earthdata Login Getting 36 granules, approx download size: 0.81 GB Credentials for the cloud provider GES_DISC are not available

thank you ver much

Jorge

@Crow901 thanks for reporting this Jorge! yes, we need to update the cloud credential endpoints and this should be fixed ASAP. I'll tag you when it's ready.

Hi Jorge, we just released v0.4.1 on pip and soon will be available in conda-forge. This version fixes the issues with GES DISC, give it a try and let me know if it works for you! @Crow901

Hi @betolink thanks for fixing this bug, I just ran into this same problem recently and was thinking it would be good to have an API method which allows us to add our own DAAC cloud provider (during runtime) in the event that one of these links changes in the future. What do you think?

That's definitely a great idea @andyPark, the endpoints come from a fixed dictionary and they surely can change.

I agree that's a good idea. It can be done now in a hacky way since everything is an object. I think you'd just need to do something like:

import earthdata.daac

earthdata.daac.DAACS.append({...})
earthdata.daac.CLOUD_PROVIDERS.append('...')

Oh I didn't realize you could hack it like that, good to know.

@betolink I am still having trouble with this API, I am running the latest version 0.4.1 and I am able to download the resources I want on when I run the script on my local machine, but when I try to download the data from S3 (using an EC2 instance on us-west-2) I get an error:

Getting 32 granules, approx download size: 18.07 GB
Accessing cloud dataset using provider: GES_DISC
Credentials for the cloud provider GES_DISC are not available
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/earthdata/store.py", line 367, in get
    self._get(granules, local_path, access, provider, threads)
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/multimethod/__init__.py", line 315, in __call__
    return func(*args, **kwargs)
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/earthdata/store.py", line 459, in _get_granules
    s3_fs.get(file, local_path)
AttributeError: 'NoneType' object has no attribute 'get'

Here is what I am running:

auth = Auth().login(strategy="netrc")

start = "2022-09-01T00:00:00.000"
end = "2022-09-09T00:00:00.000"

granule_query = DataGranules()\
    .concept_id('C2089270961-GES_DISC')\
    .bounding_box(-125,24,-66,50)\
    .temporal(start, end)
granules = granule_query.get(50)
# AWS S3
data_links = [granule.data_links(access="direct") for granule in granules]
store = Store(auth)
store.get(granules, local_path='./data')

to be clear, I am able to download these files from S3 when I do it manually using bash shell on my EC2 instance.

Thanks!

Hi @andyPark yup, there is a typo in the cloud providers name and an unrelated bug with the downloader because of this.

You can try this workaround

auth = Auth().login(strategy="netrc")

start = "2022-09-01T00:00:00.000"
end = "2022-09-09T00:00:00.000"

granule_query = DataGranules()\
    .concept_id('C2089270961-GES_DISC')\
    .bounding_box(-125,24,-66,50)\
    .temporal(start, end)
granules = granule_query.get(50)
data_links = [granule.data_links(access="direct")[0] for granule in granules]

s3fs= store.get_s3fs_session('GESDISC')
s3fs.get(data_links, "./data")

Note that the provider is GES_DISC but I added GESDISC in the dictionary (duh!) I'll work on a fix tomorrow and alas the library name will change to earthaccess this week.

Oh thanks for the work around. Yes, I thought it was odd that GESDISC was in the dictionary haha but wasn't sure if that was the real issue because I was unfamiliar with the codebase.

closing the issue as GESDISC cloud access is possible now, now the caveat is that on-prem is not available until GESDISC implements the new bearer tokens.