kubeflow
Adding securityContext in the Deployment will prevent the SparkApplications to be launched
In my env I have to run the workload with a non-root user. I've successfully configured the Spark Operator Deployment to start with a custom user ID and Group ID like bellow:
securityContext:
runAsUser: 5000
runAsGroup: 5000
$ kubectl -n nci-service-spark-dev exec -it spark-operator-fd675d579-zthsm -- id
uid=5000 gid=5000 groups=5000
The problem is that I can't start any Spark Applications if the above config is added. I tried to use the Spark Pi available in this repo: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-pi.yaml
I also tried to add the securityContext or the podSecurityContext there but it doesn't seem to start. I'm receiving the following error :
$ kubectl -n nci-service-spark-dev describe sparkapplication.sparkoperator.k8s.io/spark-pi
[...]
Status:
Application State:
Error Message: failed to run spark-submit for SparkApplication nci-service-spark-dev/spark-pi: WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "main" java.lang.IllegalArgumentException: basedir must be absolute: ?/.ivy2/local
at org.apache.ivy.util.Checks.checkAbsolute(Checks.java:48)
at org.apache.ivy.plugins.repository.file.FileRepository.setBaseDir(FileRepository.java:135)
at org.apache.ivy.plugins.repository.file.FileRepository.<init>(FileRepository.java:44)
at org.apache.spark.deploy.SparkSubmitUtils$.createRepoResolvers(SparkSubmit.scala:1166)
at org.apache.spark.deploy.SparkSubmitUtils$.buildIvySettings(SparkSubmit.scala:1261)
at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:51)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
State: FAILED
[...]
I have to mention that if I remove the securityContext part from the Spark Operator Deployment, everything works as expected. It's probably something related to the permissions but I can't find a solution or at least a workaround for it.
@marius-lupu try to do a manual build of using Dockerfile.rh which is meant for OpenShift (it enforces stricter policies and doesn't allow running as root). Just make sure you amend this line FROM golang:1.15.2-alpine as builder as it's an older one in Dockerfile.rh.
@aneagoe, thank you for your message. I tried with the image build from the Dockerfile.rh but I'm still receiving the failure posted above.
@marius-lupu The error is raised because there is no default HOME set for the user defined in the securityContext, and the ivy2 cache folder path cannot be formed correctly. As the issue here (https://stackoverflow.com/a/55921242) says, it's necessary to define the ivy2 folder using the Spark configurations.
@marius-lupu The error is raised because there is no default HOME set for the user defined in the securityContext, and the ivy2 cache folder path cannot be formed correctly. As the issue here (https://stackoverflow.com/a/55921242) says, it's necessary to define the ivy2 folder using the Spark configurations.