bitsail icon indicating copy to clipboard operation
bitsail copied to clipboard

[BitSail][Bug][StreamingFileSystemSink]PartitionCommitFunction GlobalAggregateResult class can't find with child first class loader

Open lujg opened this issue 2 years ago • 1 comments

Describe the bug

Flink uses akka rpc to update global aggregate. And akka is loaded by system class loader.

User jar will not be found in system class loader, so we'd better to serialize aggregation input params and deserialize aggregation results by ourself instead of using TaskSnapshotMeta and GlobalAggregateResult directly.

To Reproduce

We can reproduce the bug as follows (use screenshots if necessary):

  1. The job conf is: '....'

  2. The test data is: '....'

  3. Bitsail run command: '....'

  4. The error can be seen in: '....'

(Note that the above sentences are not necessary for each bug report issue.)

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Build Environment

Describe the environment where the test project was build.

  • OS [e.g. Debain 11.0]
  • JDK [e.g. 1.8.0_301]
  • Maven [e.g. 3.6.3]
  • ...

Execution Environment

Describe the environment where the test project ran.

  • OS [e.g. Debain 11.0]
  • JDK [e.g. 1.8.0_301]
  • Deployment [e.g. Yarn | CDH 6.3.x]
  • ...

Additional context

Add any other context about the problem here.

lujg avatar Jan 11 '23 09:01 lujg

@lujg Wait for your contributes, i assigned issue to you.

hk-lrzy avatar Jan 12 '23 01:01 hk-lrzy