parseable
parseable copied to clipboard
Enhancement: Introduce scalable query nodes
This PR introduces distributed, scalable querying in a stateless manner. The idea is that query nodes should be able to scale up and down like ingestor nodes. The only difference being that since these nodes are also writing data to S3, we have to have one leader to coordinate API calls and writing operations.
Description
This PR has:
- [ ] been tested to ensure log ingestion and log query works.
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added documentation for new or modified features or behaviors.
TODO:
- [ ] Thorough testing
- [ ] Migration flow validation
- [ ] Ingestor stream creation
- [ ] Pulling data from ingestor staging in a query