loki query timeout with high volume of logs

[rajeevpadinharepattu]

We have a single binary loki installation on an on-premise kuberentes cluster, where the applications push around 10GB of logs everyday.

I have created a pipeline to create a custom label as below.

pipelineStages:
  - match:
      selector: '{container="program-server"}'
      stages:
      - regex:
          expression: "(?P<events>program-loki)(?P<space> : )(?P<content>.*)"
      - output:
          source: content
      - labels:
          services: events

but while we query against the above service label, loki is showing timeout errors as below

Get "http://loki.logging:3100/loki/api/v1/query_range?direction=backward&end=1684316144208000000&limit=1000&query=%7Bservices%3D%22socket-loki-event-persist-message%22%7D+%7C%3D+%60%27EventCode%27%3A+228%60&start=1684143344208000000&step=120000ms": net/http: timeout awaiting response headers (Client.Timeout exceeded while awaiting headers)

I dont see any issues with the pipeline stages as the query works very well with less volume of logs..

Even with high volume logs , I am able to query against the program-server container with any time range (using the container label.) without any issues.

Expected behavior successfull loki queries with high volume of data

Environment: Kubernetes 1.24.7 Deployment tool: helm