EKS cross-cluster and cross-namespace syncing issue.

[jdepp]

The Main Question

Can cross-cluster and cross-namespace syncing be done with KubeD?

Our setup

• 4 EKS clusters - one cluster called cent that we'd like to sync secrets to clusters dev, stg, and prod. We had to create our own Docker image based on KubeD's image so that we could install the AWS CLI so that we could auth kubectl to our EKS clusters.
• We'd like to be able to deploy a secret in the default namespace in cent and have that sync to namespace test. We'd like for this secret to then sync to dev/stg/prod's default and test namespaces as well.
• Labeled namespace test with label name=test in all clusters.

Here is the k8s manifest for a test secret that we deploy to the default namespace in cent:

apiVersion: v1
data:
  val1: [REDACTED]
kind: Secret
metadata:
  creationTimestamp: null
  name: test-jeremy
  annotations:
    kubed.appscode.com/sync: "name=test"
    kubed.appscode.com/sync-contexts: "us-west-2-dev,us-west-2-stg,us-west-2-prod"

We find that this correctly syncs the secret to our dev, stg, and prod clusters in their respective default namespace, but does not seem to sync to any of the clusters' test namespace (including cent's). Upon further inspection, the secret does sync to the test namespace in dev/stg/prod, however it deletes the secret in test right after creating it. Also, if we specify in our Secret's manifest to also sync to our cent cluster (via kubed.appscode.com/sync-contexts: "us-west-2-cent,us-west-2-dev,us-west-2-stg,us-west-2-prod"), that it will cross cluster sync and cross namespace sync, however the Secret in the test namespace in cent also creates and then deletes right away.

Helm setup

We use Helmfile to install our Helm charts, but hopefully the values provided here will help anyone who can help us troubleshoot. As mentioned earlier, we had to create our own Docker image called kubed in our own repo with a few additional tools in order to auth to our EKS clusters.

  - name: "kubed"
    namespace: "default"
    chart: "../charts/kubed"
    installed: true
    values:
      - operator:
          registry: [REDACTED]
          repository: "kubed"
          operator.tag: "v0.12.0"
        imagePullSecrets:
          - name: "dockerconfig"
        imagePullPolicy: "Always"
        serviceAccount:
          annotations:
            eks.amazonaws.com/role-arn: "arn:aws:iam::{{ .Environment.Values.accountNumber }}:role/k8s-kubed"
        config:
          configSourceNamespace: "default"
          clusterName: {{ .Environment.Values.context }}

Additional Info

• Cross namespace syncing seems to work if we don't include cross cluster syncing configuration.
• Ideally we'd be able to cross namespace sync and cross cluster sync, but even just cross namespace syncing is very useful.
• The IAM Role used by KubeD is correctly mapped in our aws-auth ConfigMap in each cluster to have system:masters permissions in order to rule out a permissions issue.

[darkstarmv]

i have been able to setup replication of the secrets from central cluster to satellites with a dedicated secret for each cluster, not sure if same secret can be used for in cluster and cross cluster replication.

[solidnerd]

Hey @jdepp, I would be interested in the patched Dockerfile to get it to work. I think that it's easier to reproduce it.