flink icon indicating copy to clipboard operation
flink copied to clipboard

Published 20 hours ago • verified publisher icon apache

[FLINK-26182][Connector/pulsar] Create a e2e tests for Pulsar sink.

Open syhily opened this issue 2 years ago • 2 comments

What is the purpose of the change

Add e2e tests for the existing Pulsar sink connector. The tests are based on the new connector testing framework.

Brief change log

  • Drop the embedded Pulsar runtime for it's useless now.
  • Enable the transaction in the Pulsar container by shell script.
  • Extra the common test code and create common testing tool for Pulsar source and sink.
  • Refactor the Pulsar source test code by using the new testing tool.
  • Fix some typos in the comments of the Pulsar connector code.
  • Fix the bug in the Key_Shared subscription.
  • Add new e2e test for Pulsar sink.
  • Enable the e2e tests for Pulsar.
  • Disable the e2e tests for JDK 11.

Verifying this change

This change added tests and can be verified as follows:

  • Add tests for both Pulsar source or sink on the testing framework, which contains a lot of tests case.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes)
  • The serializers: (yes)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduces a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

syhily avatar Aug 15 '22 17:08 syhily

CI report:

  • 4f0919b2cd313857c94185d4595e547c925e986a Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Aug 15 '22 17:08 flinkbot

@tisonkun Can you help me review this PR?

syhily avatar Aug 15 '22 18:08 syhily

LGTM. Although I still suggest you remove FixedRangeGenerator.

FixedRangeGenerator is an old class which may be used by users. I prefer to keep it.

syhily avatar Sep 28 '22 17:09 syhily

Observe potential Pulsar transaction issue that causes unstable test cases. -1 to prevent unintentional merge.

I have changed the defaultNumberOfNamespaceBundles to 1 in Docker environment which can fix the transaction issue.

syhily avatar Sep 29 '22 13:09 syhily

@flinkbot run azure

tisonkun avatar Sep 29 '22 19:09 tisonkun

Root cause located: https://github.com/apache/pulsar/issues/17876. Workaround pushed by @syhily

tisonkun avatar Sep 29 '22 19:09 tisonkun