pglite icon indicating copy to clipboard operation
pglite copied to clipboard

`pglite-sync` saves incorrect `offset` for shapes returning more than one chunk

Open AntonOfTheWoods opened this issue 11 months ago • 2 comments

(+ some discussion on discord).

According to @thruflo , "The electric-offset header "Must be used as the value of the offset parameter in your next request." https://electric-sql.com/openapi.html#/paths/~1v1~1shape/get". pglite-sync currently uses messageAggregator[messageAggregator.length - 1].offset https://github.com/electric-sql/pglite/blob/6b60fbc55c2d59eb7642d1d3f560999ec5b4ec40/packages/pglite-sync/src/index.ts#L252

As all operation-level offsets appear to be the same, regardless of what is returned by the electric-offset header, that means pglite-sync will save the incorrect offset for any shape that returns more than one chunk initially.

What appears needs doing is to keep the last valid Offset returned by stream.lastOffset (so ignore 0_inf), which you can get at the moment you receive the last chunk.

AntonOfTheWoods avatar Jan 23 '25 13:01 AntonOfTheWoods

Thanks for finding this @AntonOfTheWoods!

samwillis avatar Jan 23 '25 13:01 samwillis

What's the right behaviour here -- is the last message offset different from the electric-offset value? If so, is that correct?

thruflo avatar Jan 23 '25 13:01 thruflo