pulsar icon indicating copy to clipboard operation
pulsar copied to clipboard

[wip][fix][fn] Add function name and id to producer name to avoid conflict

Open jiangpengcheng opened this issue 1 year ago • 3 comments

Fixes #21911

Main Issue: #xyz

PIP: #xyz

Motivation

there will be producer name conflicts error when function enable effectively_once and subscribed to a partitioned input topic

Modifications

Add functionId and instanceId to the suffix of the producer name to avoid conflicts

Verifying this change

  • [x] Make sure that the change passes the CI checks.
  • [ ]
  • [x] This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • [ ] Dependencies (add or upgrade a dependency)
  • [ ] The public API
  • [ ] The schema
  • [ ] The default values of configurations
  • [ ] The threading model
  • [ ] The binary protocol
  • [ ] The REST endpoints
  • [ ] The admin CLI options
  • [ ] The metrics
  • [ ] Anything that affects deployment

Documentation

  • [ ] doc
  • [ ] doc-required
  • [x] doc-not-needed
  • [ ] doc-complete

Matching PR in forked repository

PR in forked repository: https://github.com/jiangpengcheng/pulsar/pull/21

jiangpengcheng avatar Jan 17 '24 11:01 jiangpengcheng

Thanks for your fix. I have a question. After adding the instance name to producerName, can the effectively_once be guaranteed? For example, there was no ack for the message before the failure, but the message was resent after switching to another instance. I think deduplication is invalid at this time because of different producerNames. @jiangpengcheng

graysonzeng avatar Jan 18 '24 01:01 graysonzeng

@graysonzeng hm, you are right, let's consider an another way to fix the issue

jiangpengcheng avatar Jan 18 '24 07:01 jiangpengcheng

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (24c927e) 73.60% compared to head (247c4af) 72.39%. Report is 12 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #21912      +/-   ##
============================================
- Coverage     73.60%   72.39%   -1.21%     
+ Complexity    32359    32030     -329     
============================================
  Files          1859     1861       +2     
  Lines        138373   143467    +5094     
  Branches      15160    16278    +1118     
============================================
+ Hits         101853   103870    +2017     
- Misses        28640    31483    +2843     
- Partials       7880     8114     +234     
Flag Coverage Δ
inttests 24.28% <0.00%> (+0.11%) :arrow_up:
systests 23.67% <0.00%> (-0.06%) :arrow_down:
unittests 71.53% <75.00%> (-1.35%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...a/org/apache/pulsar/functions/sink/PulsarSink.java 73.75% <75.00%> (-0.10%) :arrow_down:

... and 113 files with indirect coverage changes

codecov-commenter avatar Jan 21 '24 15:01 codecov-commenter