seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

[draft][Spark streaming] fix spark streaming

Open laglangyue opened this issue 2 years ago • 2 comments

Purpose of this pull request

Check list

  • [ ] Code changed are covered with tests, or it does not need tests for reason:
  • [ ] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
  • [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
  • [ ] If you are contributing the connector code, please check that the following files are updated:
    1. Update change log that in connector document. For more details you can refer to connector-v2
    2. Update plugin-mapping.properties and add new connector information in it
    3. Update the pom file of seatunnel-dist
  • [ ] Update the release-note.

laglangyue avatar May 06 '23 21:05 laglangyue

great job

hailin0 avatar May 19 '23 09:05 hailin0

these days,I'm busy with a poc.

I have not fully tested the state recovery on Yarn, as there is no Yarn environment at home.

laglangyue avatar May 20 '23 02:05 laglangyue

please resolve the conflict

hailin0 avatar Jul 25 '23 14:07 hailin0

please check code style

hailin0 avatar Aug 03 '23 05:08 hailin0

I can't submit it success it on yarn.I don't know why image

laglangyue avatar Aug 10 '23 07:08 laglangyue

image image

The state has not been recovered, no commit log has been generated, the offset log is not well connected, and the source state is fully generated. There are some differences from the mechanism of Spark Structured Streaming. There may be some adaptations needed for the state in some places.

laglangyue avatar Aug 16 '23 12:08 laglangyue

hello any progress?

hailin0 avatar Sep 19 '23 07:09 hailin0