odc-tools icon indicating copy to clipboard operation
odc-tools copied to clipboard

Glob Support in S3 find is weird

Open omad opened this issue 1 year ago • 1 comments

I don't understand what globs are supported and what aren't. Wasted so much time on this.

$ s3-find s3://dea-public-data-dev/ausbath_09_v4/ga_multi_\*json 
$ s3-find s3://dea-public-data-dev/ausbath_09_v4/ga_multi_\*.json 
$ s3-find s3://dea-public-data-dev/ausbath_09_v4/\*.json 
s3://dea-public-data-dev/ausbath_09_v4/ga_multi_ausbath_0.stac-item.json

Which was made more inscrutiable by the indefinite hang (see #105 )

s3-to-dc --stac --update-if-exists --skip-check s3://dea-public-data-dev/ausbath_09_v4/ga_multi_ausbath_0.stac-item.json ga_multi_ausbath_0
/usr/local/lib/python3.8/dist-packages/datacube/drivers/postgres/_api.py:48: RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
  select([
Traceback (most recent call last):
  File "/usr/local/bin/s3-to-dc", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/s3_to_dc.py", line 169, in cli
    added, failed, skipped = dump_to_odc(
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/s3_to_dc.py", line 93, in dump_to_odc
    raise IndexingException("Doc stream was empty")
odc.apps.dc_tools.utils.IndexingException: Doc stream was empty

#### HANGS HERE ####

omad avatar Jun 09 '23 05:06 omad

Indefinite hang resolved by #562

Ariana-B avatar Jul 10 '23 03:07 Ariana-B