Writing SparkApplication yaml file for a Spring Boot web application

[wschung1113]

Hi all,

I've been struggling for the last couple of days trying to deploy a SparkApplication for our Spring Boot project. I read through the official user-guide (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#mounting-configmaps) and threads on "Issues" but am still confused about some things.

` apiVersion: "sparkoperator.k8s.io/v1beta2" kind: SparkApplication metadata: name: spark-test namespace: sparkapp spec: type: Java mode: cluster #image: "gcr.io/spark-operator/spark:v3.1.1" image: bi33-repo:31313/visualization:spark imagePullPolicy: Always mainClass: com.tmax.hyperdata.visualanalytics.VisualanalyticsApplication #mainApplicationFile: "local:home/catdragon/VisualanalyticsApplication/visualanalytics-0.0.1-SNAPSHOT.jar" #mainApplicationFile: "local:///opt/spark/jars/visualanalytics-0.0.1-SNAPSHOT.jar" sparkVersion: "3.1.1" restartPolicy: type: Never volumes: - name: "spark-pv" persistentVolumeClaim: claimName: spark-pvc driver: cores: 1 coreLimit: "1200m" memory: "512m" labels: version: 3.1.1 serviceAccount: sparktest volumeMounts: - name: "spark-pv" mountPath: "/db" envVars: HD_HOME: "/db" envFrom: - secretRef: name: spark-conf-secret #javaOptions: "-Dlog4j.configuration=/home/catdragon/VisualanalyticsApplication" executor: cores: 1 instances: 1 memory: "512m" labels: version: 3.1.1 volumeMounts: - name: "spark-pv" mountPath: "/db" envVars: HD_HOME: "/db" envFrom: - secretRef: name: spark-conf-secret #javaOptions: "-Dlog4j.configuration=/home/catdragon/VisualanalyticsApplication"

`

To start off with, here are my SparkApplication specs. Things such as spark-operator or service account has been set up. I have enabled webhook in order to overwrite some environment variables for our Spring Boot app.

1. Do I enter the image of our app or gcr.io/spark-operator/spark:v3.1.1?

• There are not a lot of examples on the web other than the pi_spark tutorial so I couldn't figure which image to specify.

2. If I must specify the location of .jar file of our app along our application image, which I think should be redundant, is the location fixed? Where should it be? Inside our image?

Above are my two main concerns. I have managed to figure out how to overwrite env variables by enabling webhook creating the spark-operator and creating a Secret file instead of a ConfigMap for environment variables.

If anybody can help me out with my concerns or guide me to an appropriate link that I can study it'd be awesome. If there is a similar issue posted please let me know and I will put down this one. Also please let me know if I can provide any additional information solving my concerns.

Thank you!