Multitenancy: RunnerDeployment not using githubAPICredentialsFrom secret

[Jalmeida1994]

Controller Version

0.25.2

Helm Chart Version

0.20.2

CertManager Version

1.9

Deployment Method

ArgoCD

cert-manager installation

We followed the documentation and it's working correctly.

Checks

• [X] This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• [X] I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• [X] My actions-runner-controller version (v0.x.y) does support the feature
• [X] I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue

Resource Definitions

---
# Source: secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: organization-github-app-secret  
  namespace: organization
data:
  github_app_id: ***
  github_app_installation_id: ***
  github_app_private_key: ***
---
# Source: hra.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: organization-autoscaler
  namespace: organization
spec:
  githubAPICredentialsFrom:
    secretRef: 
      name: organization-github-app-secret
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    name: organization-runners
  minReplicas: 0
  maxReplicas: 5
  scaleUpTriggers:
  - githubEvent: {}
    amount: 1
    duration: 30m
---
# Source: runner_deployment.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: organization-runners
  namespace: organization
spec:
  template:
    metadata:
    spec:
      githubAPICredentialsFrom:
        secretRef: 
          name: organization-github-app-secret 
      ephemeral: true
      labels:
        - organization-runners
      organization: organization
      containers:
        - name: runner
          imagePullPolicy: IfNotPresent

To Reproduce

1. Install actions-runner-controller chart v0.20.2. Added a default auth secret "controller-manager" to it;
2. Deploy RunnerDeployment, HRA and Secret with App Auth in new namespace;
3. Runner can't register with GitHub.

Describe the bug

New RunnerDeployment can't register with GitHub using a new Secret with a different GitHub Auth. Controller outputs:

2022-08-10T13:49:29Z	ERROR	actions-runner-controller.runner	Failed to get new registration token	{"runner": "organization-runners-2jw7w-bbsr8", "error": "failed to create registration token: POST https://HOSTNAME/api/v3/orgs/organization/actions/runners/registration-token: 403 Resource not accessible by integration []"}

The error seems that the provided app auth don't have permission to authenticate with the organization in GitHub, however, manually generating the JWT and installation_id using the app credentials works in getting a registration token. Moreover, when creating multiple ARC deployments, the same App credentials work with GitHub.

I also found out that the generated RunnerDeployment manifest doesn't have the githubAPICredentialsFrom: key:

$ kubectl get runnerdeployment.actions.summerwind.dev/organization-runners -n organization-oyaml 

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  creationTimestamp: "2022-08-10T13:19:16Z"
  generation: 2
  name: organization-runners
  namespace: organization
  resourceVersion: "9649260"
  uid: ab71fd50-21ea-445e-942d-764ee8d322b0
spec:
  effectiveTime: null
  replicas: 0
  selector: null
  template:
    spec:
      containers:
      - imagePullPolicy: IfNotPresent
        name: runner
        resources: {}
      dockerdContainerResources: {}
      image: ""
      labels:
      - organization-runners
      organization: organization
status:
  availableReplicas: 0
  desiredReplicas: 0
  readyReplicas: 0
  replicas: 0
  updatedReplicas: 0

Nor the kubectl describe command. I upgraded the CRDs, as the controller accepts the githubAPICredentialsFrom: field in the CRDs.

Weirder still, the HorizontalRunnerDeployment has the field in its generated manifest, as I would expect with the RunnerDeployment.

So just a theory as I just checked a little bit of the code, the RunnerDeployment/Runner is not fetching the secret from the key githubAPICredentialsFrom: but the original one deployed with the controller. (Now that I'm thinking more this shouldn't work as the secret and Runner are in different namespaces).

Describe the expected behavior

The RunnerDeployment/Runner should fetch the secret from githubAPICredentialsFrom: and register the token successfully.

Controller Logs

https://gist.github.com/Jalmeida1994/8c4e1ffca0bdef112f4b1dbfb70831f8

Runner Pod Logs

N/A

Additional Context

Really sorry if this is stupidity from me ahah. I've been stuck for a couple of hours checking this, and I still haven't found the solution.

Thanks for such an awesome job everyone. Cheers!