spark-operator icon indicating copy to clipboard operation
spark-operator copied to clipboard

sparkConf creates incorrect path from `spark.jars.packages` and `spark.jars.repositories`

Open arichardet opened this issue 5 years ago • 1 comments

Using spark-operator:v1beta2-1.2.0-3.0.0, and creating a SparkApplication as follows:

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: pyspark-pi
  namespace: spark-operator
spec:
  sparkConf:
    spark.jars.packages: "org.apache.hadoop:hadoop-aws:2.7.6,com.amazonaws:aws-java-sdk:aws-java-sdk-1.7.6"
    spark.jars.repositories: <redacted>
...

The url that is created is incorrect: <redacted>/com/amazonaws/aws-java-sdk/aws-java-sdk-1.7.6/aws-java-sdk-aws-java-sdk-1.7.6.jar Note the repeated aws-java-sdk-aws-java-sdk

arichardet avatar Oct 09 '20 17:10 arichardet

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Oct 14 '24 12:10 github-actions[bot]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

github-actions[bot] avatar Nov 03 '24 14:11 github-actions[bot]