flink icon indicating copy to clipboard operation
flink copied to clipboard

[FLINK-34123][core][type] Introduce built-in serialization support for map and lists

Open X-czh opened this issue 1 year ago • 2 comments

What is the purpose of the change

Introduce built-in serialization support for map and lists, which falls back to Kyro previously.

Brief change log

Introduce built-in serialization support for map and lists when extracting types. The serializers and de-serializers have already been defined previously.

Set is another common Java collection type, but there's no built-in serializer for it yet (MultiSetTypeInformation utilizes MapSerializer underneath, but it could be more efficient for common Set), we can add support for it in a following PR.

Verifying this change

This change added tests that validates maps and lists are serialized using the built-in serializer instead of Kyro.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

X-czh avatar Apr 08 '24 13:04 X-czh

CI report:

  • e26a8fe6647a26d94e6de128ca10704e7404eada Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Apr 08 '24 13:04 flinkbot

@reswqa Hi, could you help take a look when you have time?

X-czh avatar Apr 08 '24 14:04 X-czh

@reswqa Thanks for the review. I've rebased on latest master branch, and it has been highlighted in the doc content as follows:

Currently, only Map, List and its super interface Collection are supported.

Would you mind taking a look again when you have time?

X-czh avatar May 23 '24 04:05 X-czh

Thanks @X-czh, I think we only need two commits. One for the implementation and one for documentation.

reswqa avatar May 23 '24 05:05 reswqa

Thanks @X-czh, I think we only need two commits. One for the implementation and one for documentation.

Thanks for the notice, @reswqa. I've rearranged the commits to have only two commits: one for impl, one for doc.

X-czh avatar May 23 '24 05:05 X-czh