cortex icon indicating copy to clipboard operation
cortex copied to clipboard

Published 20 hours ago • verified publisher icon cortexproject

Distributor memory usage is high

Open jakirpatel opened this issue 3 years ago • 2 comments

Describe the bug We have on-premise cortex cluster. Currently we have 15 distributors and 15 ingesters. Currently 1 distributor is behaving weird with memory consumption reaching 100% of its limit memory.

Version Cortex 1.11.0

Expected behavior Distributor should run without OOM.

Environment:

  • Infrastructure: Kubernetes on bare metal managed through Rancher.
  • Deployment tool: Custom created with kustomize.

Storage Engine

  • [ ] Blocks - Google Block Storage.

Additional Context

  1. GOGC : We have setup GOGC values for both distributor and the ingester. The ingester does have the GOGC=30 and distributor does have the GOGC=80.

I think somehow the when the ingester GC happens we are seeing this impact. Is there any way to improve the situation.

jakirpatel avatar Jun 03 '22 06:06 jakirpatel

Hi, is your Prometheus sending HA labels? If so, this PR fixes a memory leak: https://github.com/cortexproject/cortex/pull/4739

If this is not the case, would you mind taking a heap profile?

alvinlin123 avatar Jun 10 '22 03:06 alvinlin123

FWIW updating to 1.13.0-rc.0 resolved a lot of my memory problems, as @alvinlin123 suggested.

TaylorMutch avatar Jul 10 '22 22:07 TaylorMutch

Good to know @TaylorMutch :-)

alvinlin123 avatar Aug 18 '22 17:08 alvinlin123