parseable icon indicating copy to clipboard operation
parseable copied to clipboard

Metadata management flow

Open nitisht opened this issue 3 years ago • 1 comments

Current STREAM_INFO map may have an issue with parallel writes to the map for the same stream.

  1. We need to implement a locking mechanism that allows exclusive writes to the map.
  2. Additionally Metadata map get and get methods should handle both local and S3. This way we don’t duplicate all the code to fallback on object store everywhere. a. Get Flow: Fetch metadata from local and fallback to object store if not found. b. Set Flow: Set to object store and then try setting it locally.

nitisht avatar Aug 15 '22 12:08 nitisht

List streams should fetch data from metadata first, instead of hitting S3 for each call. See #69 for context

nitisht avatar Aug 27 '22 14:08 nitisht

@trueleo is this still relevant?

nitisht avatar Nov 20 '22 02:11 nitisht

@nitisht No

trueleo avatar Nov 20 '22 02:11 trueleo