register
register copied to clipboard
PSC-STM-A3: Add Storage for PSC Stream Timepoint
When using the PSC Stream API, it requires either:
- A
timepointpointer of the last record seen through the API, which will then return records seen since here - Nothing sent through, which will obtain any new records from now
stream_client.read_stream(timepoint:)
This timepoint is important to ensure that if there is any restart of requests (eg after a timeout or failure), that no records were lost in the meantime.
This needs to be:
- Retrieved when starting up, to read latest timepoint
- Stored after a new record has been processed
A simple key/value pair is sufficient for storage and retrieval of this, so potentially Redis would be a good choice if that is available. However, note that if this is on a single EC2 instance it is a single point of failure.
Estimate: 6 hours