pgstac icon indicating copy to clipboard operation
pgstac copied to clipboard

Using Iterable with @retry leads to silent data corruption

Open Zaczero opened this issue 9 months ago • 1 comments

@retry is fundamentally incompatible with the Iterable concept.

For example here:

https://github.com/stac-utils/pgstac/blob/13b7d8d3b4d2897d5450cf92d10910d40b4c4803/src/pypgstac/src/pypgstac/load.py#L275

load_partition accepts input from itertools.groupby:

https://github.com/stac-utils/pgstac/blob/13b7d8d3b4d2897d5450cf92d10910d40b4c4803/src/pypgstac/src/pypgstac/load.py#L617-L618

That is Iterable and after first retry will contain empty data, discarding all input.

More about groupby in the Python documentation:

https://docs.python.org/3/library/itertools.html#itertools.groupby

Zaczero avatar Mar 28 '25 14:03 Zaczero