Duplicated entries when querying with filter using multi-instance local docker deployment

[nevarsin]

When deploying Loki using the infra files locate in this repo (production/docker/) the system reports a single entry up to 16 times. This does not happen when querying only with label selector. The only change I applied to the conf was setting auth_enabled to false

To Reproduce Steps to reproduce the behavior:

1. cd production/docker
2. docker compose up -d
3. punch in a sample record: curl --location --request POST 'http://localhost:8080/loki/api/v1/push' --header 'Content-Type: application/json' --data-raw '{"streams": [{"stream": {"instance": "testlogs"},"values": [["1658932457000000000","a test line"]]}]}'

If your query is {"instance":"testlogs"} then you will correctly get 1 entry If your query has filtering applied, e.g. {"instance":"testlogs"} |= "test"

Then you'll get the same record duplicated 16x

{"status":"success","data":{"resultType":"streams","result":[{"stream":{"instance":"testlogs"},"values":[["1658932457000000000","a test line"],["1658932457000000000","a test line"],["1658932457000000000","a test line"],["1658932457000000000","a test line"],["1658932457000000000","a test line"],["1658932457000000000","a test line"],["1658932457000000000","a test line"],["1658932457000000000","a test line"],["1658932457000000000","a test line"],["1658932457000000000","a test line"],["1658932457000000000","a test line"],["1658932457000000000","a test line"],["1658932457000000000","a test line"],["1658932457000000000","a test line"],["1658932457000000000","a test line"],["1658932457000000000","a test line"]]}],"stats":{"summary":{"bytesProcessedPerSecond":5602,"linesProcessedPerSecond":164,"totalBytesProcessed":1632,"totalLinesProcessed":48,"execTime":0.291322159,"queueTime":62.742899831,"subqueries":5,"totalEntriesReturned":16},"querier":{"store":{"totalChunksRef":16,"totalChunksDownloaded":16,"chunksDownloadTime":16614559,"chunk":{"headChunkBytes":0,"headChunkLines":0,"decompressedBytes":640,"decompressedLines":16,"compressedBytes":768,"totalDuplicates":16}}},"ingester":{"totalReached":3072,"totalChunksMatched":32,"totalBatches":32,"totalLinesSent":32,"store":{"totalChunksRef":0,"totalChunksDownloaded":0,"chunksDownloadTime":0,"chunk":{"headChunkBytes":0,"headChunkLines":0,"decompressedBytes":992,"decompressedLines":32,"compressedBytes":1248,"totalDuplicates":0}}}}}}

Environment:

• Local docker compose file

I suppose I'm missing something Loki config side but with this base config as a start, it's troublesome to pinpoint where the issue may lie.

[Bigouden]

Same for me with or without "auth_enabled". Without query-frontend, it works normally

Edit : Works well using pull model (https://grafana.com/docs/loki/latest/configuration/query-frontend/#grpc-mode-pull-model)

[fti7]

Same here. Workaround: loki.yaml: query_range: parallelise_shardable_queries: false