actions-runner-controller icon indicating copy to clipboard operation
actions-runner-controller copied to clipboard
verified publisher icon actions

Unable to get registration_token 404 Error (GET request instead of POST)

Open roshitho opened this issue 2 years ago • 3 comments

Checks

  • [X] I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
  • [X] I'm not using a custom entrypoint in my runner image

Controller Version

latest

Helm Chart Version

v3.11.3

CertManager Version

latest

Deployment Method

Helm

cert-manager installation

Steps in - https://github.com/actions/actions-runner-controller/blob/master/docs/quickstart.md

Checks

  • [X] This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
  • [X] I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
  • [X] My actions-runner-controller version (v0.x.y) does support the feature
  • [X] I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
  • [X] I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: example-runnerdeploy
spec:
  replicas: 1
  template:
    spec:
      repository: myorg/myrepo

To Reproduce

1. Install cert manager
2. Add PAT token
3. Helm deployment

helm upgrade --install --namespace actions-runner-system --create-namespace\
  --set=authSecret.create=true\
  --set=authSecret.github_token="ghp_xxxxxPAT"\
  --set=githubURL="https://githubesurl/api/v3"\
  --set=githubUploadURL="https://githubesurl/api/uploads"\
  --set=runnerGithubURL="https://githubesurl"\
  --set=additionalVolumeMounts[0].mountPath="/usr/local/share/ca-certificates"\
  --set=additionalVolumeMounts[0].name=ghe-cert\
  --set additionalVolumes[0].name=ghe-cert\
  --set additionalVolumes[0].secret.secretName=ghecert\
  --wait actions-runner-controller actions-runner-controller/actions-runner-controller

4. Create a runner against the repo

kubectl apply -f runnerdeployment.yaml  (example-runnerdeploy)

Expected runners in RUNNING status

Describe the bug

I am setting up arcs on Azure AKS for self-hosted runners to be used from github enterprise server (on-prem) .

I have performed the steps as in https://github.com/actions/actions-runner-controller/blob/master/docs/quickstart.md

But the pods do not come up. The controller log shows the following error- failed to create registration token: GET https://gitesurl/api/v3/repos/myorg/myrepo/actions/runners/registration-token: 404 Not Found

 2023-05-01T18:45:31Z    DEBUG   events  Updating registration token failed      {"type": "Warning", "object": {"kind":"Runner","namespace":"default","name":"github-runner-api-rq6d4-gq9c2","uid":"cd5dd01e-8213-44a8-abb3-99b2f702c2fa","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"697203"}, "reason": "FailedUpdateRegistrationToken"}
2023-05-01T18:45:31Z    ERROR   runner  Failed to get new registration token    {"runner": "github-runner-api-123-bzc6p-2mwgq", **_"error": "failed to create registration token: GET https://gitesurl/api/v3/repos/myorg/myrepo/actions/runners/registration-token: 404 Not Found []"_**}
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).updateRegistrationToken
        github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:456
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).processRunnerCreation
        github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:292
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).Reconcile
        github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:147
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235

I am expecting a POST request here not GET. Is the issue because of that?

Following steps I have had to do additionally to make it work with github enterprise.

  1. Add additional volume and volume mount for certificate (--set=additionalVolumeMounts[0].name=gh-crt ....)
  2. Add host alias in deployment.
  3. Set env for GITHUB_ENTERPRISE_URL=https://enterpriseurl
  4. set env for certificate SSL_CERT_DIR=/usr/share/ca-certificates
  5. Add additional params for step 1 (helm upgrade --install --namespace actions-runner-system --create-namespace ...) . --set=githubURL="https://githubesurl/api/v3"
    --set=githubUploadURL="https://githubesurl/api/uploads"
    --set=runnerGithubURL="https://githubesurl"\

Does any of it affect the call? I am assuming GET will not be implemented (a 405 error/unimplemented error) .

Any pointers would be very helpful.

Thanks

Describe the expected behavior

Runner in running status.

Whole Controller Logs

sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235
2023-05-02T01:48:34Z    DEBUG   events  Updating registration token failed      {"type": "Warning", "object": {"kind":"Runner","namespace":"default","name":"github-runner-api-rq6d4-gq9c2","uid":"cd5dd01e-8213-44a8-abb3-99b2f702c2fa","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"697203"}, "reason": "FailedUpdateRegistrationToken"}
2023-05-02T01:48:34Z    ERROR   runner  Failed to get new registration token    {"runner": "github-runner-api-123-bzc6p-2mwgq", "error": "failed to create registration token: GET https://githubesurl/api/v3/repos/myorg/myrepo/actions/runners/registration-token: 404 Not Found []"}
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).updateRegistrationToken
        github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:456
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).processRunnerCreation
        github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:292
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).Reconcile
        github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:147
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235
2023-05-02T01:48:34Z    DEBUG   events  Updating registration token failed      {"type": "Warning", "object": {"kind":"Runner","namespace":"default","name":"github-runner-api-123-bzc6p-2mwgq","uid":"d926654a-81f5-4cdf-85e9-4416511b1c27","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"728061"}, "reason": "FailedUpdateRegistrationToken"}
2023-05-02T01:48:48Z    INFO    runnerdeployment        Waiting until the newest runnerreplicaset to be 100% available {"runnerdeployment": "default/github-runner-api", "newest_runnerreplicaset": "default/github-runner-api-t8tjv", "newest_runnerreplicaset_replicas_ready": 0, "newest_runnerreplicaset_replicas_desired": 1, "old_runnerreplicasets_count": 2}
2023-05-02T01:48:56Z    DEBUG   runnerreplicaset        Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "default/example-runnerdeploy-h7lk7", "owner": "default/example-runnerdeploy-h7lk7-blvjz", "pods": null}
2023-05-02T01:48:56Z    DEBUG   runnerreplicaset        Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "default/github-runner-api-rq6d4", "owner": "default/github-runner-api-rq6d4-gq9c2", "pods": null}
2023-05-02T01:48:56Z    DEBUG   runnerreplicaset        Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "default/github-runner-api-123-bzc6p", "owner": "default/github-runner-api-123-bzc6p-2mwgq", "pods": null}
2023-05-02T01:48:56Z    DEBUG   runnerreplicaset        Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "default/github-runner-api-t8tjv", "owner": "default/github-runner-api-zl728-fxv8m", "pods": null}
2023-05-02T01:48:56Z    DEBUG   runnerreplicaset        Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "default/github-runner-api-zl728", "owner": "default/github-runner-api-zl728-fxv8m", "pods": null}
2023-05-02T01:48:56Z    INFO    runnerdeployment        Waiting until the newest runnerreplicaset to be 100% available {"runnerdeployment": "default/github-runner-api", "newest_runnerreplicaset": "default/github-runner-api-t8tjv", "newest_runnerreplicaset_replicas_ready": 0, "newest_runnerreplicaset_replicas_desired": 1, "old_runnerreplicasets_count": 2}

Whole Runner Pod Logs

No listener pod

Additional Context

  1. K8s is Azure AKS
  2. Github is enterprise server on-premise
  3. Provided SSL and host alias and set GITHUB_ENTERPRISE_URL and SSL_CERT_DIR
  4. Curl to POST method works form jump-host.

roshitho avatar May 02 '23 01:05 roshitho

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

github-actions[bot] avatar May 02 '23 01:05 github-actions[bot]

I noticed the same error using v0.27.4 of ARC. The problem went away when I rolled back to v0.27.3.

gtatps avatar Aug 03 '23 06:08 gtatps

For me it was the issue below

I used "spec.template.spec.repository" field instead of "spec.template.spec.organization" in the runnerDeployment, when using github app creds for an organization. when changes it to organization everything worked.

The logs are not very helpful to understand what is wrong, and pin point the issue did took some time.

Hope it will help someone.

adamyodinsky avatar Jun 12 '24 06:06 adamyodinsky