earthaccess icon indicating copy to clipboard operation
earthaccess copied to clipboard

Use generators/iterator for granule query

Open betolink opened this issue 3 years ago • 1 comments

If we are querying hundred of thousands of granules, we need to handle them in a more efficient way, perhaps implementing an iterator like

links = []
for g in Datagranules().concept_id("c-some-datasets").items():
    links.append(g.data_links())

This idea is that we'll only load the current page into memory instead of the whole number of granules in a collection query.

betolink avatar Aug 15 '22 00:08 betolink

As we've recently discussed during an earthaccess hack day, I'm going to create an issue in the python_cmr library for implementing this functionality upstream, so we don't duplicate the functionality here.

chuckwondo avatar Apr 18 '24 10:04 chuckwondo