spark-operator
spark-operator copied to clipboard
kubeflow
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
God, what am I supposed to do, thank you ```shell kubectl apply -f examples/spark-pi.yaml ``` ```java 22/04/11 09:15:45 ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: External scheduler cannot be instantiated at...
Hi all, We notice that if a `sparkapplication` is deleted immediately after creation, the driver/executor pods of this `sparkapplication` will still be created and keep running until completed. To reproduce,...
Hi I am relatively new to spark and I am trying to implement it on on-prem infrastructure. The spark job gets submitted succesfully but runs only with the driver pod....
I am trying to build a `sparkctl` CLI tool locally but failed to do so, following [this](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/sparkctl/README.md) guide. The error message is ``` # golang.org/x/sys/unix ../../../go/pkg/mod/golang.org/x/[email protected]/unix/syscall_darwin.1_13.go:29:3: //go:linkname must refer to...
Hello, Using image `gcr.io/spark-operator/spark-operator:v1beta2-1.2.1-3.0.0` I encountered a weird problem when using Airflow's [SparkKubernetesOperator](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/_api/airflow/providers/cncf/kubernetes/operators/spark_kubernetes/index.html) and [SparkKubernetesSensor](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/_api/airflow/providers/cncf/kubernetes/sensors/spark_kubernetes/index.html). In TL;DR, the SparkKubernetesOperator calls 2 subsequent API calls to the spark-operator - `Delete`, and...
Copy of PR https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/pull/1386 That PR was closed because one mistake when I rebased master
``` ++ id -u + myuid=0 ++ id -g + mygid=0 + set +e ++ getent passwd 0 + uidentry=root:x:0:0:root:/root:/bin/bash + set -e + '[' -z root:x:0:0:root:/root:/bin/bash ']' + '['...
How to generate Java crd codes using https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/charts/spark-operator-chart/crds/sparkoperator.k8s.io_sparkapplications.yaml ? I tried everything, all failed.
Running Spark 3.1.1 on spark-operator with Istio 1.5.7 and Ipv6 environment. After submitting a job I am getting below exception: Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Invalid proxy server configuration at...
