register icon indicating copy to clipboard operation
register copied to clipboard

PSC-STM-A3: Add Storage for PSC Stream Timepoint

Open tiredpixel opened this issue 1 year ago • 0 comments

When using the PSC Stream API, it requires either:

  • A timepoint pointer of the last record seen through the API, which will then return records seen since here
  • Nothing sent through, which will obtain any new records from now
stream_client.read_stream(timepoint:)

This timepoint is important to ensure that if there is any restart of requests (eg after a timeout or failure), that no records were lost in the meantime.

This needs to be:

  • Retrieved when starting up, to read latest timepoint
  • Stored after a new record has been processed

A simple key/value pair is sufficient for storage and retrieval of this, so potentially Redis would be a good choice if that is available. However, note that if this is on a single EC2 instance it is a single point of failure.

Estimate: 6 hours

tiredpixel avatar Feb 28 '24 16:02 tiredpixel