odc-tools
odc-tools copied to clipboard
Glob Support in S3 find is weird
I don't understand what globs are supported and what aren't. Wasted so much time on this.
$ s3-find s3://dea-public-data-dev/ausbath_09_v4/ga_multi_\*json
$ s3-find s3://dea-public-data-dev/ausbath_09_v4/ga_multi_\*.json
$ s3-find s3://dea-public-data-dev/ausbath_09_v4/\*.json
s3://dea-public-data-dev/ausbath_09_v4/ga_multi_ausbath_0.stac-item.json
Which was made more inscrutiable by the indefinite hang (see #105 )
s3-to-dc --stac --update-if-exists --skip-check s3://dea-public-data-dev/ausbath_09_v4/ga_multi_ausbath_0.stac-item.json ga_multi_ausbath_0
/usr/local/lib/python3.8/dist-packages/datacube/drivers/postgres/_api.py:48: RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
select([
Traceback (most recent call last):
File "/usr/local/bin/s3-to-dc", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/s3_to_dc.py", line 169, in cli
added, failed, skipped = dump_to_odc(
File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/s3_to_dc.py", line 93, in dump_to_odc
raise IndexingException("Doc stream was empty")
odc.apps.dc_tools.utils.IndexingException: Doc stream was empty
#### HANGS HERE ####
Indefinite hang resolved by #562