parseable
parseable copied to clipboard
Metadata management flow
Current STREAM_INFO map may have an issue with parallel writes to the map for the same stream.
- We need to implement a locking mechanism that allows exclusive writes to the map.
- Additionally Metadata map get and get methods should handle both local and S3. This way we don’t duplicate all the code to fallback on object store everywhere. a. Get Flow: Fetch metadata from local and fallback to object store if not found. b. Set Flow: Set to object store and then try setting it locally.
List streams should fetch data from metadata first, instead of hitting S3 for each call. See #69 for context
@trueleo is this still relevant?
@nitisht No