updating httpPort or sqlPort does not update `cockroachdb-public` service port

[taroface]

I deployed a cluster with httpPort: 8081 and was able to access the DB Console on this port. I subsequently changed this back to httpPort: 8080 and this successfully performed a rolling restart of the pods with the new port. However, when running kubectl port-forward service/cockroachdb-public 8080, I got this error:

error: Service cockroachdb-public does not have a service port 8080

Running kubectl get service shows:

cockroachdb-public    ClusterIP   10.3.242.84   <none>        26258/TCP,8081/TCP,26257/TCP   24m

Is there another step that needs to update the service to use the new port?

Note: I was still able to port-forward to 8081, but this quickly threw errors like

E0514 16:50:16.364820   63081 portforward.go:400] an error occurred forwarding 8081 -> 8081: error forwarding port 8081 to pod c86376879927c9d62242ca840a9a4c4d6427fee3b5974e84586e47ac2d88643f, uid : exit status 1: 2021/05/14 20:50:16 socat[94227] E connect(5, AF=2 127.0.0.1:8081, 16): Connection refused

as expected.

This is using the Operator version currently at https://github.com/cockroachdb/cockroach-operator/blob/master/manifests/operator.yaml on GKE and CRDB v20.2.8.

[keith-mcclellan]

This also applies to changing the SQL port - we need to update the services when we change ports on the pod configs. @chrislovecnm @udnay We need to get this prioritized ASAP. Is there a workaround (maybe a kubectl patch service ) command that would work or is this a release blocker?

[keith-mcclellan]

I have a workaround - if you delete ${clustername}-public eg oc delete service crdb-tls-example-public --namespace=cockroachdb after updating the CR with the new port, when the operator recreates the service it gets generated with the new port.

@udnay @taroface I suggest we document this as a known issue with a workaround, and then backlog this issue to be fixed after the release. Thoughts?

fwiw, I also tested manually patching the service, but that was more steps so I think this is cleaner.

[taroface]

@keith-mcclellan @udnay Can we use a red callout where the doc describes changing the port, or would you prefer to have this on the Known Limitations page? https://www.cockroachlabs.com/docs/v21.1/known-limitations.html

[taroface]

Added a callout to this file in the open PR: https://github.com/cockroachdb/docs/pull/10578/files

[keith-mcclellan]

we can close when https://github.com/cockroachdb/docs/pull/10578 is merged

[keith-mcclellan]

@keith-mcclellan @udnay Can we use a red callout where the doc describes changing the port, or would you prefer to have this on the Known Limitations page? https://www.cockroachlabs.com/docs/v21.1/known-limitations.html

I think we should call this out in the doc specifically. Can you host this somewhere for a final review @taroface ? I think it looks good but I want to read it formatted. Thanks!

[taroface]

@keith-mcclellan Posted in https://github.com/cockroachdb/docs/pull/10578, but the preview is here: https://deploy-preview-10578--cockroachdb-docs.netlify.app/docs/v21.1/operate-cockroachdb-kubernetes.html

[keith-mcclellan]

Callout LGTM thanks @taroface !

[chrislovecnm]

@keith-mcclellan is this a release blocker?

[chrislovecnm]

@keith-mcclellan i am removing release blocker

[chrislovecnm]

@udnay ^

[udnay]

It's not a release blocker, we can remove the label and put it into the backlog.