kubeflow
FailedMount Configmap not found warning on spark driver pod
I'm running spark operator on kubeadm. and got the error:
Warning FailedMount 7m26s kubelet MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-0251af7c7dfbe657-conf-map" not found
Hello,
I'm having the same warning, but I still haven't identified what it is!!!
MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-5d2ffa7c88dec4ff-conf-map" not found
However, the process ends successfully.
I believe this is expected if you're not also running the mutating admission webhook. Check out the README for what it's used for. Can you confirm whether or not you're running that?
I'm having the same warning with the webhook running. it looks like the config map is created 2s after it has failed to load on the driver.
I built spark operator docker image from cloned repo and pushed to my local artifactory with tag 1.3.1-3.1.1. after deployed operator, could see deployment results like here
deployment.apps/sparkoperator created clusterrole.rbac.authorization.k8s.io/spark-mutating-webhook created clusterrolebinding.rbac.authorization.k8s.io/spark created role.rbac.authorization.k8s.io/spark-role created rolebinding.rbac.authorization.k8s.io/spark created serviceaccount/spark created job.batch/sparkoperator-init created service/spark-operator-webhook created
then spark operator log shows below:
root:x:0:0:root:/root:/bin/bash echo root:x:0:0:root:/root:/bin/bash [[ -z root:x:0:0:root:/root:/bin/bash ]] exec /usr/bin/tini -s -- /usr/bin/spark-operator -v=2 -logtostderr -namespace=spark-operator -enable-ui-service=true -ingress-url-format= -controller-threads=10 -resync-interval=30 -enable-batch-scheduler=false -label-selector-filter= -enable-metrics=true -metrics-labels=app_type -metrics-port=10254 -metrics-endpoint=/metrics -metrics-prefix= -enable-webhook=true -webhook-svc-namespace=spark-operator -webhook-port=8080 -webhook-timeout=30 -webhook-svc-name=sparkoperator-webhook -webhook-config-name=sparkoperator-webhook-config -webhook-namespace-selector= -enable-resource-quota-enforcement=true I0112 12:50:57.252870 10 main.go:145] Starting the Spark Operator I0112 12:50:57.253588 10 main.go:178] Enabling metrics collecting and exporting to Prometheus I0112 12:50:57.253848 10 metrics.go:142] Started Metrics server at localhost:10254/metrics I0112 12:50:57.255734 10 watcher.go:43] Creating new resource usage watcher I0112 12:50:57.356252 10 webhook.go:219] Starting the Spark admission webhook server I0112 12:50:57.365166 10 webhook.go:424] Creating a MutatingWebhookConfiguration for the Spark pod admission webhook I0112 12:50:57.394247 10 webhook.go:452] Creating a ValidatingWebhookConfiguration for the SparkApplication resource quota enforcement webhook I0112 12:50:57.411905 10 main.go:220] Starting application controller goroutines I0112 12:50:57.412003 10 controller.go:165] Starting the workers of the SparkApplication controller I0112 12:50:57.412361 10 controller.go:97] Starting the ScheduledSparkApplication controller I0112 12:50:57.412398 10 controller.go:103] Starting the workers of the ScheduledSparkApplication controller
after I deploy my spark-sream app, driver not mounting configmap, but configmap actually exists.
This issue resolved in below spark operator image. Start using this image.
ghcr.io/googlecloudplatform/spark-operator:v1beta2-1.3.4-3.1.1
From: yangqi766 @.> Sent: Monday, May 30, 2022 10:31 AM To: GoogleCloudPlatform/spark-on-k8s-operator @.> Cc: 7amarivu @.>; Comment @.> Subject: Re: [GoogleCloudPlatform/spark-on-k8s-operator] FailedMount Configmap not found warning on spark driver pod (#1368)
how to resolve this warning ?
— Reply to this email directly, view it on GitHubhttps://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/1368#issuecomment-1140699015, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AXIVRPLPP235NRQVVVFCS6TVMRDQ3ANCNFSM5F7DYISA. You are receiving this because you commented.Message ID: @.***>
I use the version is v1beta2-1.3.6-3.1.1 ! And I also test the version v1beta2-1.3.4-3.1.1 , but the warning still exists .
Is custom confgmap mounted in both versions? if config volume mounted to driver and executor pod, can ignore this warning.
From: yangqi766 @.> Sent: Monday, May 30, 2022 10:56 AM To: GoogleCloudPlatform/spark-on-k8s-operator @.> Cc: 7amarivu @.>; Comment @.> Subject: Re: [GoogleCloudPlatform/spark-on-k8s-operator] FailedMount Configmap not found warning on spark driver pod (#1368)
I use the version is v1beta2-1.3.6-3.1.1 ! And I also test the version v1beta2-1.3.4-3.1.1 , but the warning still exists .
— Reply to this email directly, view it on GitHubhttps://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/1368#issuecomment-1140713804, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AXIVRPKTZ75WQYXF642MYHTVMRGQ7ANCNFSM5F7DYISA. You are receiving this because you commented.Message ID: @.***>