quickwit

quickwit copied to clipboard

### Description I added the parameter to limit the size of the query string in a user input. Further, I moved `validate_request` out of the for loop since many of...

kamalesh0406

Add more control plane metrics: - number of shards assigned with locality. - number of plan change - etc.

fulmicoton

Indexing twice the same file with the local ingest command generate worrying log errors

1

Everything is going well, but we log some errors, and we don't know if quickwit indexed something or not. ```bash quickwit tool local-ingest --index hdfs-logs --input-path hdfs-logs-multitenants-10000.json --config ../config/quickwit.yaml ❯...

fmassot

(in reality it should finish it works?) ``` No more indexing pipeline on this index and source, killing merge pipeline. index_id=simian_chico_1434753389610517491 source_id=_ingest-source", ``` INFO 2024-01-16T07:09:48.674817923Z [labels."k8s-pod.statefulsetKubernetesIo.pod-name": simian-indexer-2] [resource.labels.containerName: quickwit] 2024-01-16T07:09:48.674Z...

fulmicoton

includes cardinality aggregation and term aggregation perf improvement for large "size" parameters The store size increase in the gh test is coming from the additional optional index in the multivalue...

PSeitz

Add Queue source with SQS implementation

2

### Description This PR proposes the generic implementation of a "queue" source. For now, only an implementation for AWS SQS with its data backed by AWS S3 is exposed to...

rdettai

Optimize json parsing for validation

1

fulmicoton

Ideally we would like to have a decent picture of the memory usage between - ephemeral cache - fast field cache - hotcache - buffers if relevant ?

fulmicoton

Connect shard throughput

1

Tested on simian with a pathologic workload: ``` - chico(rate=40mb) - chico(rate=10kb)*10 ``` On the left, without any per-shard weighting. On the right, with per-shard weighting. indexing throughput  cpu...

fulmicoton

Add support for the cardinality aggregation.

5

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-cardinality-aggregation.html I am not sure what the right approach should be here. Do we want an approximate sketch for cardinality that supports union, or do we just want to carry...

fulmicoton