spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard

[FEA] use dist/pom files as source of truth for spark versions

Open pxLi opened this issue 3 years ago • 1 comments

Is your feature request related to a problem? Please describe. snapshot and released spark shims versions list should take dist/pom file as source of truth instead of define in multiple places, e.g. we hard code in https://github.com/NVIDIA/spark-rapids/blob/branch-22.10/jenkins/version-def.sh#L52 list of tests https://github.com/NVIDIA/spark-rapids/blob/branch-22.10/jenkins/spark-premerge-build.sh#L43-L54

build and test scripts would be nice to relying on the config at https://github.com/NVIDIA/spark-rapids/blob/branch-22.10/dist/pom.xml#L43-L60, e.g. pre-merge nightly version-def ... scripts

pxLi avatar Mar 30 '22 00:03 pxLi

cc @YanxuanLiu to help check this issue. could refer to https://github.com/NVIDIA/spark-rapids/blob/branch-22.10/.github/workflows/mvn-verify-check.yml#L45-L47 of how to get pom property and format the output

pxLi avatar Aug 08 '22 07:08 pxLi