spark-operator
spark-operator copied to clipboard
kubeflow
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
I am experiencing an intermittent issue with the Spark Operator's monitoring feature, specifically when it's configured to expose metrics to Prometheus. Occasionally, the Spark driver fails to start due to...
To move forward with a legacy application, I need to pull `gcr.io/spark-operator/spark:v3.1.1` image It seems that image does not exist anymore ``` ❯ docker pull gcr.io/spark-operator/spark:v3.1.1 ``` Maybe somebody removed...
There is a pull request addressing this issue (#1685) but nobody has reviewed it.
I have some jobs running on spark operator but some of the jobs staying in "PENDING_RERUN" for several days, Is there any limit on how many jobs spark operator can...
HI, I'm using `spark-on-k8-operator` to run multiple spark applications on the same Kubernetes cluster at the same time. I have a couple of spark job that runs simultaneously and I...
We want to use persistentvolumeclaim as local storage for shuffling the data in the executors. We have tried the options given in the below link but the executors are not...
This PR fixes a context leak bug in your code. ## Summary While triaging your project, our bug fixing tool generated the following message- > In file: [event.go](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/sparkctl/cmd/event.go#L157), method `context.WithTimeout`...
Does anyone have this problem ? ``` $k logs pods/manual-simple-spark-application-driver ++ id -u + myuid=8000 ++ id -g + mygid=100 + set +e ++ getent passwd 8000 + uidentry=ailake:x:8000:100::/home/ailake:/bin/bash +...
Spark's kubernetes executor [uses the same value for memory request and memory limit](https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L212), and the current operator API matches that: although we have both `cores` and `coreLimit`, there is only...
The line [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/hack/gencerts.sh#L162] has cat expression of the secret keys that prints the whole TLS secret and ends up in the pod log of the webhook init container.
