Dynatrace operator failing to fetch the Token to rollout the oneagent installation in kubernets

[vivuu1989]

We tried to setup AKS monitoring using the dynatrace by following classic full stack installation.

1. we created dynatrace operator using kubectl apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.7.0/kubernetes.yaml

2)We created A access Token with all the defined scope permissions and created the secret using the command

kubectl -n dynatrace create secret generic dynatracesecret --from-literal="apiToken=dtxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

3. Applied the dynakube crd with below configurations enabled.

apiVersion: dynatrace.com/v1beta1
kind: DynaKube
metadata:
  name: mydynatrace
  namespace: dynatrace
spec:
  apiUrl: https://xxxxxx.dynatrace-managed.com/1xxxxxxxxxxxxxx/api
  tokens: dynatracesecret
  skipCertCheck: false
  networkZone: azurezone

  oneAgent:
    classicFullStack:
      image: ""

      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists

  activeGate:
    # Enables listed ActiveGate capabilities
    capabilities:
      - routing
      - kubernetes-monitoring
      - dynatrace-api

    image: ""

    resources:
      requests:
        cpu: 500m
        memory: 512Mi
      limits:
        cpu: 1000m
        memory: 1.5Gi

But the oneagent rollout is not happening and we could see the below error The error stating as below lastTransitionTime: "2022-08-09T19:39:54Z" message: 'error when querying token on secret dynatrace:xxxxxxx: invalid character ''<'' looking for beginning of value'

I decrypted the secret key and cross verified the API token, which is correct and not sure why this error is coming. Due to this the oneagent nodes are still not rolled out in the cluster.

{"level":"info","ts":"2022-08-09T19:55:54.663Z","logger":"dynakube-controller","msg":"problem with token detected","dynakube":"xxxxxxx","token":"APIToken","msg":"error when querying token on secret dynatrace:xxxxx: invalid character '<' looking for beginning of value"} {"level":"info","ts":"2022-08-09T19:55:54.663Z","logger":"dynakube-status","msg":"updating DynaKube CR","cause":"token conditions updated","dynakube":"xxxxxxxx"}

Note: - we have Istio as service mesh on top of kubernetes.