Request: more context in errors reported to logger, specifically 'Unknown topic or partition'

[apeloquin-agilysys]

Environment Information

• OS: Mac M3 Sonoma 14.6.1
• Node Version: 20.14.0
• NPM Version: 10.7.0
• confluent-kafka-javascript version: 1.0.0

Steps to Reproduce

In general the errors sent to the logger often lack some critical context that would make the source easy to identify.

In our case we have an application that is spinning up consumers for dozens of topics from different components. The following is an example error for a recent issue as received in the logger:

{
  "message": "Consumer encountered error while consuming. Retrying. Error details: KafkaJSProtocolError: Broker: Unknown topic or partition : Error: Broker: Unknown topic or partition\n    at Function.createLibrdkafkaError [as create] (/Users/peloquina/src/agilysys-inc/stay/backplane-base/node_modules/@confluentinc/kafka-javascript/lib/error.js:459:10)\n    at /Users/peloquina/src/agilysys-inc/stay/backplane-base/node_modules/@confluentinc/kafka-javascript/lib/kafka-consumer.js:557:29\n    at callbackTrampoline (node:internal/async_hooks:130:17)",
  "extras": {
    "name": "Confluent#consumer-6",
    "fac": "BINDING",
    "timestamp": 1736870741813
  }
}

If the extras included the topic name in this case, it would have saved hours spent narrowing down which consumer was actually at fault.

The name value seems like it should be a useful cross-reference, but as it a # private field on the consumer it can only be associated with other log entries and is completely inaccessible from within our application.

[milindl]

Regarding correlation between the consumer and the logs, the name within the consumer is a concatenation of the config's clientId and consumer-$N or producer-N. By setting the clientId, it's possible to recognize which consumer is logging the statement.

        kafkaJS: {
            // ...
            clientId: 'myConsumer'
        },

Will have logs like

...
  timestamp: 1737687965299,
  name: 'myConsumer#producer-1'
}

Regarding the other question, I'll check if it's possible to add the topic/partition info within the message.

[apeloquin-agilysys]

Hi @milindl, we are already using client ID when creating the Kafka class instances. However, when that same class is used to create dozens of consumers for different topics this doesn't do much to help narrow it down.