parseable icon indicating copy to clipboard operation
parseable copied to clipboard

Enhancement: Introduce scalable query nodes

Open parmesant opened this issue 1 year ago • 0 comments

This PR introduces distributed, scalable querying in a stateless manner. The idea is that query nodes should be able to scale up and down like ingestor nodes. The only difference being that since these nodes are also writing data to S3, we have to have one leader to coordinate API calls and writing operations.

Description


This PR has:

  • [ ] been tested to ensure log ingestion and log query works.
  • [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • [ ] added documentation for new or modified features or behaviors.

TODO:

  • [ ] Thorough testing
  • [ ] Migration flow validation
  • [ ] Ingestor stream creation
  • [ ] Pulling data from ingestor staging in a query

parmesant avatar Nov 18 '24 06:11 parmesant