aws-cloud-map-mcs-controller-for-k8s

aws-cloud-map-mcs-controller-for-k8s copied to clipboard

We use cloudmap controller in multiple clusters and have the same problem on them. Clusters are linked by vpc peering. Half of the time, an imported service cannot be reached in a pod.

• Service name is correctly resolved in an internal kube adress.
• If I use the endpointslice adress of the remote service, it works on every pod.
• I created a DaemonSet for testing purposes to see if it can be a network/node problem. A pod that have worked on a node can fail after a restart and vice versa.
• We used mcs-controller in v0.2.3 and updated in v0.3.0 and have the same problems.
• Kubernetes 1.21 on all clusters.

I have a feeling that there is some trouble between the service and endpointslice resolution but it don't know how can I debug it.

Hello @ThomasBoutelier - Which version of coredns are you using?

Also, if you can share the logs, below is the command

kubectl logs -f -l control-plane=controller-manager -c manager -n cloud-map-mcs-system

Hello,

It's the version v1.8.4.

Here's the output of the logs:

{"level":"info","ts":1664418556.359755,"logger":"controllers.ServiceExport","msg":"no changes to export to Cloud Map","namespace":"bidder-es-inte","name":"matcher-es-http"}
{"level":"info","ts":1664423785.0764976,"logger":"controllers.ServiceExport","msg":"updating Cloud Map service","namespace":"bidder-inte","name":"redis-master"}
{"level":"info","ts":1664423785.0765486,"logger":"cloudmap","msg":"fetching a service","namespace":"bidder-inte","name":"redis-master"}
{"level":"info","ts":1664423785.0766342,"logger":"controllers.ServiceExport","msg":"no changes to export to Cloud Map","namespace":"bidder-inte","name":"redis-master"}
{"level":"info","ts":1664423785.076691,"logger":"controllers.ServiceExport","msg":"updating Cloud Map service","namespace":"bidder-es-inte","name":"matcher-es-http"}
{"level":"info","ts":1664423785.0766985,"logger":"cloudmap","msg":"fetching a service","namespace":"bidder-es-inte","name":"matcher-es-http"}
{"level":"info","ts":1664423785.076791,"logger":"controllers.ServiceExport","msg":"no changes to export to Cloud Map","namespace":"bidder-es-inte","name":"matcher-es-http"}
{"level":"info","ts":1664423785.0768418,"logger":"controllers.ServiceExport","msg":"updating Cloud Map service","namespace":"bidder-inte","name":"iasp"}
{"level":"info","ts":1664423785.0768507,"logger":"cloudmap","msg":"fetching a service","namespace":"bidder-inte","name":"iasp"}
{"level":"info","ts":1664423785.0769007,"logger":"controllers.ServiceExport","msg":"no changes to export to Cloud Map","namespace":"bidder-inte","name":"iasp"}

Hey - I can help you troubleshoot. Can you join our slack channel? https://github.com/aws/aws-cloud-map-mcs-controller-for-k8s#slack-community

Should be fixed by the https://github.com/aws/aws-cloud-map-mcs-controller-for-k8s/issues/254.