redpanda-data
console (subchart): incorrectly sets zero valued connect configs
Hi everyone, first of all, i just want to say thanks for the cool service.
Deployed by helm chart Redpanda appVersion: v24.2.2 Redpanda console appVersion: v2.4.6
I have almost default values on it, except ingress values for console, and custom docker image for connectors. Dockerfile:
FROM redpandadata/connectors:v1.0.26
LABEL version=mongodb-kafka-connect:v1.0.5
COPY mongo-kafka-connect-1.13.0-all.jar /opt/kafka/redpanda-plugins/mongodb-kafka-connect/mongo-kafka-connect-1.13.0-all.jar
After helm install redpanda -n redpanda, everything is up
k get pods -n redpanda
NAME READY STATUS RESTARTS AGE
redpanda-0 2/2 Running 0 118m
redpanda-1 2/2 Running 0 118m
redpanda-2 2/2 Running 0 118m
redpanda-connectors-65b5595784-wtqkt 1/1 Running 1 (118m ago) 118m
redpanda-console-8598665998-x9zww 1/1 Running 0 115m
When i open Connectors tab in Redpanda console, i see available connectors.
After I create any of the Mongodb connectors, it is created, but it is not in the interface. The logs in pod show the following error
│ {"level":"warn","ts":"2024-10-14T17:05:19.753Z","msg":"failed to list connectors from Kafka connect cluster","cluster_name":"connectors","cluster_address":"http://redpanda-connectors.re │
│ dpanda.svc.cluster.local:8083","error":"Get \"http://redpanda-connectors.redpanda.svc.cluster.local:8083/connectors?expand=info&expand=status\": context deadline exceeded"} │
│ {"level":"warn","ts":"2024-10-14T17:05:19.917Z","msg":"failed to list connectors from Kafka connect cluster","cluster_name":"connectors","cluster_address":"http://redpanda-connectors.re │
│ dpanda.svc.cluster.local:8083","error":"Get \"http://redpanda-connectors.redpanda.svc.cluster.local:8083/connectors?expand=info&expand=status\": context deadline exceeded"} │
│ {"level":"warn","ts":"2024-10-14T17:06:14.084Z","msg":"failed to list connectors from Kafka connect cluster","cluster_name":"connectors","cluster_address":"http://redpanda-connectors.re │
│ dpanda.svc.cluster.local:8083","error":"Get \"http://redpanda-connectors.redpanda.svc.cluster.local:8083/connectors?expand=info&expand=status\": context deadline exceeded"} │
│ {"level":"warn","ts":"2024-10-14T17:13:17.970Z","msg":"failed to list connectors from Kafka connect cluster","cluster_name":"connectors","cluster_address":"http://redpanda-connectors.re │
│ dpanda.svc.cluster.local:8083","error":"Get \"http://redpanda-connectors.redpanda.svc.cluster.local:8083/connectors?expand=info&expand=status\": context deadline exceeded"} │
│ {"level":"warn","ts":"2024-10-14T17:13:18.138Z","msg":"failed to list connectors from Kafka connect cluster","cluster_name":"connectors","cluster_address":"http://redpanda-connectors.re │
│ dpanda.svc.cluster.local:8083","error":"Get \"http://redpanda-connectors.redpanda.svc.cluster.local:8083/connectors?expand=info&expand=status\": context deadline exceeded"}
But when I go into Redpanda console pod and do curl http://redpanda-connectors.redpanda.svc.cluster.local:8083/connectors?expand=info&expand=status, the list of connectors is visible.
/app $ curl http://redpanda-connectors.redpanda.svc.cluster.local:8083/connectors?expand=info&expand=status
/app $ {"mongodb-source-connector-xz6j":{"info":{"name":"mongodb-source-connector-xz6j","config":{"connector.class":"com.mongodb.kafka.connect.MongoSourceConnector","publish.full.document.only":"false","output.schema.infer.value":"false","startup.mode":"latest","topic.creation.default.partitions":"1","connection.password":"my_password","output.format.value":"json","publish.full.document.only.tombstone.on.delete":"false","connection.username":"my_user","output.format.key":"json","topic.creation.enable":"true","key.converter.schemas.enable":"true","database":"my_database","connection.uri":"mongodb://my_host:27017/my_database?replicaSet=rs&ssl=false","name":"mongodb-source-connector-xz6j","topic.creation.default.replication.factor":"-1","value.converter.schemas.enable":"true","connection.url":"mongodb://my_host:27017/my_database?replicaSet=rs&ssl=false","post.processor.chain":"com.mongodb.kafka.connect.sink.processor.DocumentIdAdder","value.converter":"org.apache.kafka.connect.storage.StringConverter","key.converter":"org.apache.kafka.connect.storage.StringConverter"},"tasks":[{"connector":"mongodb-source-connector-xz6j","task":0}],"type":"source"}}}
Can you please tell me where I made a mistake?
JIRA Link: K8S-406
Hey @arthurfedorov ,
could you show us the Kafka connect config in Console? I think I've seen a similar case where a timeout of 0 was used which meant the request immediately times out as soon as it was sent.
Hello @weeco, thank you for your answer I don't really understand what configs I need to show, can you tell me exactly what I need? I went through the values.yaml helm chart and didn't find any mention of timeouts
You can also describe the console pod and provide a copy of your mounted console config. It's a YAML file and the path to that file is either provided as an env variable or command line flag. Do not forget to redact sensitive configs
Config and describe fairly basic
k describe pod -n billing redpanda-console-58b444c78c-2j4lk
Name: redpanda-console-58b444c78c-2j4lk
Namespace: billing
Priority: 0
Service Account: redpanda-console
Node: cl1ilufmcc6ui3t4diop-udec/10.30.10.37
Start Time: Wed, 16 Oct 2024 18:28:51 +0300
Labels: app.kubernetes.io/instance=redpanda
app.kubernetes.io/name=console
pod-template-hash=58b444c78c
Annotations: checksum-redpanda-chart/config: 8f92a2ce2476078fd80fbe21cd990fe140df1ea781ee463d6f17e2ee284ae7aa
checksum/config: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
cni.projectcalico.org/containerID: 363959ded925e068cffd44e742bc0582b5308c01a57d0260d95c95b03138ece8
cni.projectcalico.org/podIP: 10.112.157.109/32
cni.projectcalico.org/podIPs: 10.112.157.109/32
Status: Running
IP: 10.112.157.109
IPs:
IP: 10.112.157.109
Controlled By: ReplicaSet/redpanda-console-58b444c78c
Containers:
console:
Container ID: containerd://82aea047b4f29a8169c8a2ccf8d4741eb6ea662952cfa865e3bf7e8c742bd93d
Image: docker.redpanda.com/redpandadata/console:v2.4.6
Image ID: docker.redpanda.com/redpandadata/console@sha256:e4cb11f01eebdf013a638353d962c86f75b5046dfc427539542ed768d7104499
Port: 8080/TCP
Host Port: 0/TCP
Args:
--config.filepath=/etc/console/configs/config.yaml
State: Running
Started: Wed, 16 Oct 2024 18:37:27 +0300
Ready: True
Restart Count: 0
Liveness: http-get http://:http/admin/health delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:http/admin/health delay=10s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/console/configs from configs (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gwssq (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
configs:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: redpanda-console
Optional: false
kube-api-access-gwssq:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
/app $ cat /etc/console/configs/config.yaml
# from .Values.console.config
connect:
clusters:
- name: connectors
password: ""
tls:
caFilepath: ""
certFilepath: ""
enabled: false
insecureSkipTlsVerify: false
keyFilepath: ""
token: ""
url: http://redpanda-connectors.billing.svc.cluster.local:8083
username: ""
connectTimeout: 0
enabled: true
readTimeout: 0
requestTimeout: 0
kafka:
brokers:
- redpanda-0.redpanda.billing.svc.cluster.local.:9093
- redpanda-1.redpanda.billing.svc.cluster.local.:9093
- redpanda-2.redpanda.billing.svc.cluster.local.:9093
sasl:
enabled: false
schemaRegistry:
enabled: true
tls:
caFilepath: ""
certFilepath: ""
enabled: false
insecureSkipTlsVerify: false
keyFilepath: ""
urls:
- http://redpanda-0.redpanda.billing.svc.cluster.local.:8081
- http://redpanda-1.redpanda.billing.svc.cluster.local.:8081
- http://redpanda-2.redpanda.billing.svc.cluster.local.:8081
tls:
caFilepath: ""
certFilepath: ""
enabled: false
insecureSkipTlsVerify: false
keyFilepath: ""
redpanda:
adminApi:
enabled: true
tls:
caFilepath: ""
certFilepath: ""
enabled: false
insecureSkipTlsVerify: false
keyFilepath: ""
urls:
- http://redpanda.billing.svc.cluster.local.:9644
connectTimeout: 0 is the problem. See my responses above. Please raise an issue in the helm repository to clarify why the defaults are faulty: https://github.com/redpanda-data/helm-charts
Although commented out our docs currently provide an example of the following properties set: https://docs.redpanda.com/current/console/config/configure-console/
# Used to test cluster connectivity.
#connectTimeout: 15s
# Overall REST timeout.
#readTimeout: 60s
# Timeout for REST requests.
#requestTimeout: 6s
This was also uncovered through by a user in our community slack channel: https://redpandacommunity.slack.com/archives/C01AJDUT88N/p1730246418007929
Oh interesting, this looks like it's coming from the way that redpanda subcharts console (which directly imports the Console's config types)
@weeco could you give me a bit of education about how console's config loading works?
I'm guessing that koanf.UnmarshalConf is doing things much differently from json.Unmarshal. From what I can garner, the distinction between present and not present isn't encoded into the struct itself and instead is handled by koanf.UnmarshalConf.
We expected it to follow standard go unmarshalling behaviors 😅... The right answer might be running genpartial against the console types (which can also help with the yaml vs json tag issue). WDYT @RafalKorepta ?
The right answer might be running genpartial against the console types (which can also help with the yaml vs json tag issue). WDYT @RafalKorepta ?
Agree with that solution. That should work with any external function call.
