redpanda-data
Edit consumer group: custom values
There is a need to change the offset, for example by skipping 1, 2 or 3+ messages. Timestamp search is not accurate. Similar to https://github.com/redpanda-data/console/issues/346
Suggestion: manual change of values with custom values for consumer group and topic.
My suggestion is to be able to specify any offset value, not based on the current one.
I'm planning on contributing. @weeco would a PR for this be considered?
From a first look, seems like the back end HTTP PATCH /consumer-groups/{groupId} (that the front end calls for other offset editing options)
https://github.com/redpanda-data/console/blob/512798dfd326f4e577a8442391408686836a046a/backend/pkg/api/handle_consumer_group.go#L135
already handles custom offsets (in the case of the "Other Consumer Group" option, specific custom values are supplied), so no back end changes would be required.
Another thing to think about: how would the "Custom offset" option work with multiple partitions? I'd imagine that specifying just a single value that will be applied to all partitions would suffice, because otherwise we'd need some kind of value per partition input. If you'd want to set multiple partitions to different values, you could just go one by one (selecting "Edit" on each partition UI row)
And lastly, should there be a warning if setting the offset value outside of the watermark range?
@BenasB Good thoughts and PRs are always welcome!
I think you are right about the existing endpoint and you are asking the right questions in regards to how we should handle multiple partitions from a user perspective. Personally I'd say that it's more likely that users want to change the offset for individual partitions (see mentioned usecase of skipping messages) so that they can skip specific messages (e.g. because somehow a broken message was published and this trips up a consumer).
However I think we can do both, since we have both options too:
- Adding partition offsets (by clicking the edit icon in a specific partition)
- Editing topic offsets (which will be harder because we have to handle the edgecases such as "one out of your 9 partitions does not have the offset you provided").
I think if we just add the option to the editing partition dialogue this should address the request here. What do you think?
I'm not entirely sure I understood your
we have both options too
proposal.
Adding partition offsets
Are you sure you meant "adding" since AFAIK there's no way right now to add completely new offsets, only the ability to edit existing offsets.
Just so we're on the same page, as I understand it, there are 3 ways of editing offsets:
- Directly on partition (changes offset for 1 partition in 1 topic)
- On a topic (changes offset for n partitions in 1 topic)
- On the whole consumer group (changes offset for n partitions in m topics)
All of these render the same UI component: https://github.com/redpanda-data/console/blob/c13ca56332699762726d3202ece3c65d6f0e6da7/frontend/src/components/pages/consumers/Modals.tsx#L58
So we can't (easily) add the "custom offset" option just for "1. Directly on partition" (?)
What I was alluding to in the first place is something like the following:
and then for one partition it would be (don't mind the missing "Offset Before", it's a bug):
and for multiple partitions ("2. On a topic", "3. On the whole consumer group"):
I meant setting sorry! Your screenshots look good to me. We only need to make sure that we can't set an offset that is higher than the max watermark for that partition (or handle that case similarly as we do today when editing a whole topic offsets where we then show a warning message)
Please express your desire for this feature by reacting to this issue using the thumbs-up emoji rather than commenting here. We are planning to improve the consumer groups page and working on mocks where we can incorporate this feature. The implementation is not planned and thus I can't give an ETA yet