docker pull fails behind the corporate proxy

[vineeth-asml]

Checks

• [X] I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• [X] I'm not using a custom entrypoint in my runner image

Controller Version

0.27.4

Helm Chart Version

0.23.3

CertManager Version

0.23.3

Deployment Method

Helm

cert-manager installation

cert-manage works fine and was installed correcly

Checks

• [X] This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• [X] I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• [X] My actions-runner-controller version (v0.x.y) does support the feature
• [X] I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• [X] I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

kind: RunnerDeployment
metadata:
  name: dependabot-runner-deployment
  namespace: actions-runner-system
spec:
  replicas: 10
  template:
    metadata:
    labels:
       version: main-2023-05-05T13-29-06
    spec:
      containers:
        - env:
            - name: HTTP_PROXY
              valueFrom:
                secretKeyRef:
                  key: http_proxy
                  name: gh-proxy
            - name: HTTPS_PROXY
              valueFrom:
                secretKeyRef:
                  key: http_proxy
                  name: gh-proxy
            - name: NO_PROXY
              valueFrom:
                secretKeyRef:
                  key: no_proxy
                  name: gh-proxy
            - name: http_proxy
              valueFrom:
                secretKeyRef:
                  key: http_proxy
                  name: gh-proxy
            - name: https_proxy
              valueFrom:
                secretKeyRef:
                  key: http_proxy
                  name: gh-proxy
            - name: no_proxy
              valueFrom:
                secretKeyRef:
                  key: no_proxy
                  name: gh-proxy
          image: >-
            ghcr.io/actions/actions-runner-controller/actions-runner-dind:v2.299.1-ubuntu-20.04-aa6dab5
          name: runner
      dockerdWithinRunnerContainer: true
      labels:
        - dependabot-dont-use
      organization: my-gh

To Reproduce

1. RunnerDeployment with dockerdWithinRunnerContainer: true and ghcri.o.com/actions/actions-runner-controller/actions-runner-dind:v2.299.1-ubuntu-20.04-aa6dab5
2. Runner Deployment has the proxy configured in the deployment as we run it behind corporate firewall
3. dockerd is started correctly 
4. docker pull ubuntu

Describe the bug

docker pull from dockerhub / gcr.io etc fails in the action runner is failing with a timeout error.

runner@copy-dependabot-qddrn-dhtxw:/$ docker pull ubuntu
Using default tag: latest
Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Looks like dockerd doesn't respect the proxy settings in the RunnerDeployment file. ( note that other operations like downloading from a public site works fine ).

Same works when the rootless image is used and when docker is started as a sidecar.

Tried with a custom entrypoint as well- with latest version of docker and proxy settings in the

Describe the expected behavior

docker pull from public repositories works by consuming the proxy settings.

Whole Controller Logs

https://gist.github.com/vineeth-asml/56df9612c4da218ed6f450562fd0ec32

[ Note that it has logs from relevant period and will contain other runner deployment file logs as well ]

Whole Runner Pod Logs

https://gist.github.com/vineeth-asml/776f8a0c55b2304103773f1108b140c3

Additional Context

We are facing this issue in the dind runner with dockerdWithinRunnerContainer: true. But works in rootless and docker sidecar versions

[github-actions[bot]]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[devonhk]

I think you would have to build a custom dind image that modifies this line to set the HTTPS_PROXY / HTTP_PROXY env variables.

EDIT

Tried with a custom entrypoint as well

I could be wrong; it seems like you already tried using a custom entry point

[vineeth-asml]

I think you would have to build a custom dind image that modifies this line to set the HTTPS_PROXY / HTTP_PROXY env variables.

EDIT

Tried with a custom entrypoint as well

I could be wrong; it seems like you already tried using a custom entry point

Hey Devon. Yes, I tried rewriting the entry point with proxy details in the daemon.json as well. That didn't help.

[hbsan]

Hi, Is there any update on this issue?

[stwehrli]

Same problem here. Anyone solved this?