The consumer does not consume all partitions fairly

[syahiaoui]

Environment Information

• OS [CentOS Linux]:
• Node Version [12 and 14]:
• NPM Version [6.14.5]:
• node-rdkafka version [2.9.0]:

node-rdkafka Configuration Settings I use non-following mode consumer and each batch contains 2000 records, consumer configuration:

{
  groupId: 'frg',
  bootstrapServer: '10.120.1.20:9092',
  topic: 'rdo-stocks-v1',
  clientId: 'rdo-stock-quantity',
  enableAutoCommit: true,
  enableAutoOffsetStore: true,
  socketKeepaliveEnable: 'true',
  securityProtocol: 'SASL_PLAINTEXT',
  saslMechanisms: 'SCRAM-SHA-512',
  saslUsername: 'rdUser',
  saslPassword: 'secret',
  consumerMaxPollInterval: '200000',
  consumerAutoOffsetReset: 'earliest',
  consumerCommitInterval: 500,
  consumerPartitionAssignmentStrategy: 'roundrobin',
  debug: 'fetch',
  consumerHeartbeatInternalMs: 10000,
  consumerSessionTimeout: 20000,
  consumerFetchMessageSizeByte: 1048576,
  messageMaxBytes: 1548576,
  consumerQueuedMaxMessagesKbytes: 1048576,
  consumerMaxPollRecords: 2000,
  consumerDefaultTimeout: 100
}

UV_THREADPOOL_SIZE is set to 30. The consumer run on in a docker container and each consumer runs on a different host (8 cores).

Additional context

The consumer consumes messages well, but as we can see in the picture below, each Kafka broker is the leader of two partitions and the consumer consumes only 1 partition:

Initially, the consumer commit messages manually, I changed it to auto-commit and i have the same result. #180 Note We have a Kafka cluster with 4 brokers, and each one has only 2 cores(to this consumer we add a sink and source connector), is that enough?

Any help is welcome

Thank's Samir

[syahiaoui]

I think that the problem is related to this fixed issue on librdkafka, but it require to change C++ wrapper in node-rdkafka to support the type of per-partition polling. https://github.com/edenhill/librdkafka/issues/1395#issuecomment-384184096

@iradul can you please add the label help wanted

Thank's Samir

[martijnimhoff]

Perhaps a bit outdated, but I've submitted a PR to add support for per-partition polling: https://github.com/Blizzard/node-rdkafka/pull/1068