quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

Published 20 hours ago • verified publisher icon quickwit-oss

Improve assignement of kafka source pipelines on several indexers

Open fmassot opened this issue 1 year ago • 4 comments

A user reported the following issue (I checked the cluster state):

  • He has around 50 Kafka sources. The biggest Kafka source has 4 desired pipelines, most of which have one.
  • He has 10 indexers, each with 4-6 pipelines. 1 indexer has all the pipelines of the biggest Kafka source.

This is bad because, as you can imagine, only 1 indexer is responsible for indexing the biggest Kafka source. And all other indexers are doing less work.

fmassot avatar Feb 26 '24 13:02 fmassot

so the merge are messing things up? Should we consider this a duplicate of #4470

fulmicoton avatar Feb 26 '24 13:02 fulmicoton

Replied offline by @fmassot : no the problem is an unbalanced load on the different nodes.

fulmicoton avatar Feb 27 '24 08:02 fulmicoton