spark-rapids
spark-rapids copied to clipboard
[FEA] use dist/pom files as source of truth for spark versions
Is your feature request related to a problem? Please describe. snapshot and released spark shims versions list should take dist/pom file as source of truth instead of define in multiple places, e.g. we hard code in https://github.com/NVIDIA/spark-rapids/blob/branch-22.10/jenkins/version-def.sh#L52 list of tests https://github.com/NVIDIA/spark-rapids/blob/branch-22.10/jenkins/spark-premerge-build.sh#L43-L54
build and test scripts would be nice to relying on the config at https://github.com/NVIDIA/spark-rapids/blob/branch-22.10/dist/pom.xml#L43-L60, e.g. pre-merge nightly version-def ... scripts
cc @YanxuanLiu to help check this issue. could refer to https://github.com/NVIDIA/spark-rapids/blob/branch-22.10/.github/workflows/mvn-verify-check.yml#L45-L47 of how to get pom property and format the output