spark-operator
spark-operator copied to clipboard
kubeflow
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Application dependencies added using the `deps.packages` seem to be ignored. It should work according to the [docs](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#specifying-application-dependencies) but other users have mentioned in this issue [#352](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/352) that they are facing...
I am running some pyspark workflows. When they finish I call sc.stop() and see that in the logs the following is printed: ``` 22/09/03 00:37:17 INFO SparkUI: Stopped Spark web...
K8s provide dynamic admission control for mutation and validation. Webhook in spark operator only receives admission request about creating pod. https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/94775cd89ca0158e869fac39e1b097d1bf56a7e8/pkg/webhook/webhook.go#L363-L372 If pod in admission request is not spark pod,...
Configuring `driver.podName` causes Pod already exists error when the job gets triggered second time. One way to solve this (as per the discussion in the Slack channel) is, in case...
Currently the ScheduledSparkApplication creates a next run on the next window of the cron configuration, could it be possible to have an annotation that tells the controller to trigger a...
use kubectl apply -f manifest/spark-operator-with-webhook-install/spark-operator-with-webhook.yaml enable webhook error 9 main.go:209] MutatingWebhookConfiguration.admissionregistration.k8s.io "spark-webhook-config" is invalid: [webhooks[0].sideEffects: Required value: must specify one of None, NoneOnDryRun, webhooks[0].admissionReviewVersions: Required value: must specify one of...
l use sparkConfigMap to mout spark-defaults.conf, in this file l set this: spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog spark.sql.catalog.spark_catalog.type hive spark.sql.catalog.hive_test org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.hive_test.type hive spark.sql.catalog.hive_test.uri thrift://ip:port but these parameters are not valid。...
hi, @liyinan926 I tried to mount the configmap to the sparkapplication, but it didn't work, according to this one: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#mounting-configmaps I have enabled webhook! show my sparkapplication.yaml. ```yaml apiVersion: sparkoperator.k8s.io/v1beta2...
