pgstac
pgstac copied to clipboard
Using Iterable with @retry leads to silent data corruption
@retry is fundamentally incompatible with the Iterable concept.
For example here:
https://github.com/stac-utils/pgstac/blob/13b7d8d3b4d2897d5450cf92d10910d40b4c4803/src/pypgstac/src/pypgstac/load.py#L275
load_partition accepts input from itertools.groupby:
https://github.com/stac-utils/pgstac/blob/13b7d8d3b4d2897d5450cf92d10910d40b4c4803/src/pypgstac/src/pypgstac/load.py#L617-L618
That is Iterable and after first retry will contain empty data, discarding all input.
More about groupby in the Python documentation:
https://docs.python.org/3/library/itertools.html#itertools.groupby