seldon-core
seldon-core copied to clipboard
SeldonIO
unmarshalerDecoder: unknown field "useClientProtocol" in seldon controller
Describe the bug
We have a problem with the usage of Seldon Operator version 1.13.1. Operator was installed to be used with Istio, that in our cluster is at version 1.10.6. When we create the SeldonDeployment object, the operator can't proceed with the creation of Istio resources and this error happens:
E0608 07:08:41.418999 1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha3.DestinationRule: failed to list *v1alpha3.DestinationRule: v1alpha3.DestinationRuleList.Items: []v1alpha3.DestinationRule: v1alpha3.DestinationRule.v1alpha3.DestinationRule.Spec: unmarshalerDecoder: unknown field "useClientProtocol" in v1alpha3.ConnectionPoolSettings_HTTPSettings, error found in #10 byte of ...|":true}}}}},{"apiVer|..., bigger context ...|nectionPool":{"http":{"useClientProtocol":true}}}}},{"apiVersion":"networking.istio.io/v1alpha3","ki|...
In the namespace where the SeldonDeployment is created we have other DestinationRules of Istio that uses the useClientProtocol field.
For what we have understood, when the operator will perform the watch of DestinationRules and it can’t do the unmarshalling of the existent ones, it doesn’t know about the exsistence of useClientProtocol and the above error happens.
We have also tried to delete the existent DestinationRules and the Seldon operator worked as expected.
Expected behaviour
It should work and complete the creation process and the seldon controller should not have this error
Environment
Environment: Google Cloud
Kubernetes version:
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.11-gke.1100", GitCommit:"20da4c21b3a6b1a56ff6ad5ecb7dee013aaf1b83", GitTreeState:"clean", BuildDate:"2022-04-01T09:40:07Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}
Seldon Images:
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata":
{
"annotations":
{},
"labels":
{
"app": "seldon",
"app.kubernetes.io/instance": "seldon-core",
"app.kubernetes.io/name": "seldon-core-operator",
"app.kubernetes.io/version": "1.13.1",
"control-plane": "seldon-controller-manager"
},
"name": "seldon-controller-manager",
"namespace": "seldon-system"
},
"spec":
{
"replicas": 1,
"selector":
{
"matchLabels":
{
"app": "seldon",
"app.kubernetes.io/instance": "seldon1",
"app.kubernetes.io/name": "seldon",
"app.kubernetes.io/version": "v0.5",
"control-plane": "seldon-controller-manager"
}
},
"template":
{
"metadata":
{
"annotations":
{
"prometheus.io/scrape": "true",
"sidecar.istio.io/inject": "false"
},
"labels":
{
"app": "seldon",
"app.kubernetes.io/instance": "seldon1",
"app.kubernetes.io/name": "seldon",
"app.kubernetes.io/version": "v0.5",
"control-plane": "seldon-controller-manager"
}
},
"spec":
{
"containers":
[
{
"args":
[
"--enable-leader-election",
"--webhook-port=4443",
"--create-resources=$(MANAGER_CREATE_RESOURCES)",
"--log-level=$(MANAGER_LOG_LEVEL)",
"--leader-election-id=$(MANAGER_LEADER_ELECTION_ID)",
""
],
"command":
[
"/manager"
],
"env":
[
{
"name": "MANAGER_LEADER_ELECTION_ID",
"value": "a33bd623.machinelearning.seldon.io"
},
{
"name": "MANAGER_LOG_LEVEL",
"value": "INFO"
},
{
"name": "WATCH_NAMESPACE",
"value": ""
},
{
"name": "RELATED_IMAGE_EXECUTOR",
"value": ""
},
{
"name": "RELATED_IMAGE_STORAGE_INITIALIZER",
"value": ""
},
{
"name": "RELATED_IMAGE_SKLEARNSERVER",
"value": ""
},
{
"name": "RELATED_IMAGE_XGBOOSTSERVER",
"value": ""
},
{
"name": "RELATED_IMAGE_MLFLOWSERVER",
"value": ""
},
{
"name": "RELATED_IMAGE_TFPROXY",
"value": ""
},
{
"name": "RELATED_IMAGE_TENSORFLOW",
"value": ""
},
{
"name": "RELATED_IMAGE_EXPLAINER",
"value": ""
},
{
"name": "RELATED_IMAGE_MOCK_CLASSIFIER",
"value": ""
},
{
"name": "MANAGER_CREATE_RESOURCES",
"value": "true"
},
{
"name": "POD_NAMESPACE",
"valueFrom":
{
"fieldRef":
{
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "CONTROLLER_ID",
"value": ""
},
{
"name": "AMBASSADOR_ENABLED",
"value": "true"
},
{
"name": "AMBASSADOR_SINGLE_NAMESPACE",
"value": "false"
},
{
"name": "PREDICTIVE_UNIT_HTTP_SERVICE_PORT",
"value": "9000"
},
{
"name": "PREDICTIVE_UNIT_GRPC_SERVICE_PORT",
"value": "9500"
},
{
"name": "PREDICTIVE_UNIT_DEFAULT_ENV_SECRET_REF_NAME",
"value": ""
},
{
"name": "PREDICTIVE_UNIT_METRICS_PORT_NAME",
"value": "metrics"
},
{
"name": "ISTIO_ENABLED",
"value": "true"
},
{
"name": "KEDA_ENABLED",
"value": "false"
},
{
"name": "ISTIO_GATEWAY",
"value": "istio-system/seldon-gateway"
},
{
"name": "ISTIO_TLS_MODE",
"value": ""
},
{
"name": "USE_EXECUTOR",
"value": "true"
},
{
"name": "EXECUTOR_CONTAINER_IMAGE_AND_VERSION",
"value": "docker.io/seldonio/seldon-core-executor:1.13.1"
},
{
"name": "EXECUTOR_CONTAINER_IMAGE_PULL_POLICY",
"value": "IfNotPresent"
},
{
"name": "EXECUTOR_PROMETHEUS_PATH",
"value": "/prometheus"
},
{
"name": "EXECUTOR_SERVER_PORT",
"value": "8000"
},
{
"name": "EXECUTOR_CONTAINER_USER",
"value": "8888"
},
{
"name": "EXECUTOR_CONTAINER_SERVICE_ACCOUNT_NAME",
"value": "default"
},
{
"name": "EXECUTOR_SERVER_METRICS_PORT_NAME",
"value": "metrics"
},
{
"name": "EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT",
"value": "http://default-broker"
},
{
"name": "EXECUTOR_REQUEST_LOGGER_WORK_QUEUE_SIZE",
"value": "10000"
},
{
"name": "EXECUTOR_REQUEST_LOGGER_WRITE_TIMEOUT_MS",
"value": "2000"
},
{
"name": "DEFAULT_USER_ID",
"value": "8888"
},
{
"name": "EXECUTOR_DEFAULT_CPU_REQUEST",
"value": "500m"
},
{
"name": "EXECUTOR_DEFAULT_MEMORY_REQUEST",
"value": "512Mi"
},
{
"name": "EXECUTOR_DEFAULT_CPU_LIMIT",
"value": "500m"
},
{
"name": "EXECUTOR_DEFAULT_MEMORY_LIMIT",
"value": "512Mi"
},
{
"name": "DEPLOYMENT_NAME_AS_PREFIX",
"value": "false"
}
],
"image": "docker.io/seldonio/seldon-core-operator:1.13.1",
"imagePullPolicy": "IfNotPresent",
"name": "manager",
"ports":
[
{
"containerPort": 4443,
"name": "webhook-server",
"protocol": "TCP"
},
{
"containerPort": 8080,
"name": "metrics",
"protocol": "TCP"
}
],
"resources":
{
"limits":
{
"cpu": "500m",
"memory": "300Mi"
},
"requests":
{
"cpu": "100m",
"memory": "200Mi"
}
},
"volumeMounts":
[
{
"mountPath": "/tmp/k8s-webhook-server/serving-certs",
"name": "cert",
"readOnly": true
}
]
}
],
"securityContext":
{
"runAsUser": 8888
},
"serviceAccountName": "seldon-manager",
"terminationGracePeriodSeconds": 10,
"volumes":
[
{
"name": "cert",
"secret":
{
"defaultMode": 420,
"secretName": "seldon-webhook-server-cert"
}
}
]
}
}
}
}
value: docker.io/seldonio/seldon-core-executor:1.13.1
image: docker.io/seldonio/seldon-core-operator:1.13.1
Is this at startup of the manager pod? at present we use istio.io/api/networking/v1alpha3
Is this at startup of the manager pod? at present we use istio.io/api/networking/v1alpha3
HI @cliveseldon , this happens during the creation of the SeldonDeployment.
The manager itself is up and running, it takes in charge the creation of the deployment and then fails due to the above error
There are Gets like here so would be good to see the full stack trace or work out where its failing. I would assume it should just get istio resources created by Seldon controller but it may be that istio operator defaults these to upgrade to a newer spec. But you are saying its only failing on DestinationRules NOT created by the controller in the same namespace?
There are Gets like here so would be good to see the full stack trace or work out where its failing. I would assume it should just get istio resources created by Seldon controller but it may be that istio operator defaults these to upgrade to a newer spec. But you are saying its only failing on DestinationRules NOT created by the controller in the same namespace?
So in the controller manager logs we get only that log.
I can confirm to you that if we remove from the namespace the DestinationRule with useClientProtocol then it works.
I would say that it breaks here more or less https://github.com/SeldonIO/seldon-core/blob/a1e8695409b423a2fa14f0becef42fe3d5a86deb/operator/controllers/seldondeployment_controller.go#L1060
because the VirtualServices part is done (there are logs in controller)