apache
[VL] Add uniffle integration
What changes were proposed in this pull request?
Add uniffle integration
How was this patch tested?
Integration test with uniffle manually When uniffle 0.8.0 release in maven, add test in gluten action flow
Thanks for opening a pull request!
Could you open an issue for this pull request on Github Issues?
https://github.com/oap-project/gluten/issues
Then could you also rename commit message and pull request title in the following format?
[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}
See also:
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Integration test with uniffle manually
gluten-celeborn has a GHA action job here. Is it possible for gluten-uniffle to add a similar one? Another option is to add some UTs, at least cover the newly added code.
@summaryzb The org.apache.spark.shuffle.UniffleColumnarBatchSerializer class is missing in the commit?
UniffleColumnarBatchSerializer
Is it possible to reuse ColumnarBatchSerializer somehow ?
Is it possible to reuse
ColumnarBatchSerializersomehow ?
Yes, it's almost the same. Change it to reuse ColumnarBatchSerializer
@PZD-CHINA @PHILO-HE we may need to also add Uniffle installation in docker, similar as Celeborn https://github.com/oap-project/gluten/blob/main/tools/gluten-te/ubuntu/dockerfile-buildenv#L118
@PZD-CHINA @PHILO-HE we may need to also add Uniffle installation in docker, similar as Celeborn https://github.com/oap-project/gluten/blob/main/tools/gluten-te/ubuntu/dockerfile-buildenv#L118
@summaryzb, suppose uniffle-0.8.0 should be pre-installed in CI docker. Please let us know if there is any other dependency required. Thanks!
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.