[Feature] consumer tag

[imjyz]

Is Your Feature Request Related to a Problem?

No

Describe the Solution You'd Like

The consumer client supports defining a consumer-tag and consumer-tag-filter-conditions.

• consumer-tag: The label defined by the consumer client for itself, not to the message. Different consumers within the same consumer-group can be configured with different tags.

• consumer-tag-filter-conditions: Filtering criteria defined by the consumer client based on message attributes, determining which messages can match the current consumer-tag; this is different from the filtering expressions used in subscription.

Constraints

• Each consumer is allowed to have only one tag (consumer-tag).

• There must be a one-to-one correspondence between consumer-tag and filter-conditions; within the same consumer group, it is strictly prohibited to have one consumer-tag associated with multiple filter-conditions.

• Consumers without a configured tag will be treated as having the default tag.

• Only pop consumers are supported.

Changes on the broker side

When a consumer pulls messages, it can only retrieve messages that match its consumer-tag.

Fallback

Perhaps we can incorporate the fallback configuration into the subscription. When fallback is allowed, messages that fail to match a specific consumer-tag will be consumed by the consumer with the default tag; when fallback is not allowed, unmatched messages can be routed to a dead-letter queue.

Note📢

The subscription remains unchanged; the consumer-tag is only used to determine which specific consumer within the consumer group will consume the messages delivered to the group.

Scenario without consumer-tag

Scenario with consumer-tag

Describe Alternatives You've Considered

Reference: https://github.com/apache/rocketmq/issues/8468

This approach requires producers to cooperate in tagging, involves heavier publishing overhead, and cannot flexibly support multiple environments.

Additional Context

In RocketMQ, all consumers under a consumer group evenly consume messages from the subscription. However, in certain scenarios, it is desired for different consumers to consume different messages (e.g., multi-environment deployments, canary releases, etc.).

Although this can be achieved by creating separate consumer groups, it introduces additional operational overhead and lacks flexibility, making it unable to support fallback scenarios.

[imjyz]

I think the requirement described in https://github.com/apache/rocketmq/wiki/RIP-8-Consumer-RACK-Support can be implemented more flexibly using consumer-tag

[imjyz]

@RongtongJin Can you help me take a look at this issue?

[RongtongJin]

This is an interesting proposal! However, it involves significant changes on both the broker and client sides. To move forward, we recommend submitting a RocketMQ Improvement Proposal (RIP) following the process outlined here: https://github.com/apache/rocketmq/wiki/RocketMQ-Improvement-Proposal.

In the RIP, you can provide a comprehensive design that clearly addresses the following aspects (include, but not be limited to):

• API/Interface Changes: Detail what new client-side APIs or configuration options will be introduced (e.g., for specifying consumer-tag and consumer-tag-filter-conditions), and how they integrate with existing subscription and consumption logic.

• Broker-Side Message Routing: Explain how the broker will manage message delivery to multiple consumers within the same consumer group, each with distinct consumer-tag and associated filter conditions. Specifically, describe how the broker ensures no message loss and maintains correct delivery semantics when multiple tagged consumers concurrently pull messages.

• Fallback and DLQ Handling: Formalize the fallback mechanism (e.g., default tag consumption vs. DLQ routing) and how it’s configured and enforced on the broker.

• Backward Compatibility: Ensure the proposal maintains compatibility with existing consumers that do not use consumer tags.

Given the complexity and cross-cutting nature of this feature, a well-structured RIP will help the community evaluate feasibility, consistency with RocketMQ’s architecture, and potential edge cases. Looking forward to your detailed proposal!

[imjyz]

The wiki isn't directly editable by me, so I've posted the draft here temporarily: https://github.com/imjyz/rocketmq/blob/develop/docs/en/RIP-84-Consumer-Tag.md