loki icon indicating copy to clipboard operation
loki copied to clipboard

Published 20 hours ago • verified publisher icon grafana

bug： ingester OOM，memory leak

Open liguozhong opened this issue 2 years ago • 4 comments

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

  1. Started Loki (SHA or version)
  2. Started Promtail (SHA or version) to tail '...'
  3. Query: {} term

Expected behavior A clear and concise description of what you expected to happen.

Environment:

  • Infrastructure: [e.g., Kubernetes, bare-metal, laptop]
  • Deployment tool: [e.g., helm, jsonnet]

Screenshots, Promtail config, or terminal output If applicable, add any output to help explain your problem. image

liguozhong avatar Jul 28 '22 09:07 liguozhong

@liguozhong can you tell us what version of loki are you using?. And loki config would be nice.

kavirajk avatar Jul 28 '22 11:07 kavirajk

@liguozhong can you tell us what version of loki are you using?. And loki config would be nice.

We are using this version. https://github.com/grafana/loki/pull/5455

I have been tracking this issue for 14 days. I am still tracking it. My job is to operate loki cluster. If I find the reason, I will continue to provide feedback in this issue.

liguozhong avatar Jul 28 '22 14:07 liguozhong

that is memory profile. ingester6-1 ingester6-2 ingester6-3 ingester6-4 ingester6-5 ingester27

liguozhong avatar Jul 28 '22 14:07 liguozhong

Thanks for details @liguozhong. Can this be related to your PR changes? https://github.com/grafana/loki/pull/5455 (I understand your loki built on top of that PR). I see this is still un-merged.

Can you confirm if you facing same issue on tip of main branch or any other released loki versions?

kavirajk avatar Jul 29 '22 08:07 kavirajk