Improve Github action for use on self-hosted runners

[kuhnroyal]

Please describe your feature request. I am using the action a lot on self-hosted runners and as a result have literally thousands of Docker images laying around. This happens due to the usage of the Dockerfile which will produce a new image on every use. On Github hosted runners this is not a problem as everything is thrown away afterwards.

Describe the solution you'd like It would be great if the action can switch to using a Docker image directly. That would mean either publishing a 2nd image or adding the entrypoint.sh to the main image.

[mikefarah]

Sorry I don't understand - how do you get thousands of Docker files?

[kuhnroyal]

Every time the Github action is used, it builds a new unnamed image and runs it. After it is done the image remains on self-hosted runners. Actually it is the same image. It just gets tagged with some random values, resulting in an endless amount of different tags for the same image.

This problem is not specific to this action, it happens for a lot of container actions running on self-hosted runners. But it can be worked around.

[mikefarah]

Not sure how github actions work under the hood - perhaps you could run the image with --rm to remove the container when it finishes?

My understanding of Docker images is that they are all built on layers on top of each other - either with the 'FROM' at the start, or the subsequent lines underneath. I'm not convinced putting entrypoint.sh into a single Dockerfile would solve this issue - that and it would affect everyone using the base docker image outside of Github Actions.

[kuhnroyal]

Currently the action is run using the Dockerfile here: https://github.com/mikefarah/yq/blob/master/github-action/Dockerfile using:

runs:
  using: 'docker'
  image: 'github-action/Dockerfile'
  args:
    - ${{ inputs.cmd }}

Instead we could add the https://github.com/mikefarah/yq/blob/master/github-action/entrypoint.sh directly to the official image at some path, lets say /github/entrypoint.sh and use the following action config:

runs:
  using: 'docker'
  image: 'mikefarah/yq:4'
  entrypoint: '/github/entrypoint.sh' 
  args:
    - ${{ inputs.cmd }}

This would also skip the whole build step in the workflow because the image is already ready to use. And there is no change to the official image except it additionally contains the entrypoint.sh file, which should not be a problem.

[kuhnroyal]

Hmm yea that works but currently there is no way to override the user in the action...

[kuhnroyal]

Needs https://github.com/actions/runner/issues/1711

[mikefarah]

I see - so it was building a new image every time from the Dockerfile. Why are you not using uses as documented here https://mikefarah.gitbook.io/yq/usage/github-action ? Does that only work with cloud github runners?

Found this on stackoverflow (and added a link to my github action docs) on setting the user: https://stackoverflow.com/questions/58955666/how-to-set-the-docker-user-in-github-actions

I haven't actually tried it though - I'd be keen to know if it works for you

[kuhnroyal]

I see - so it was building a new image every time from the Dockerfile. Why are you not using uses as documented here mikefarah.gitbook.io/yq/usage/github-action ? Does that only work with cloud github runners?

I am using it this way. What I described above is happening in that case because of the configuration in https://github.com/mikefarah/yq/blob/master/action.yml

So to recap how this works: Using uses: mikefarah/yq@xx checks out this repo at the defined branch/tag/sha, looks at the actions.yml, builds the github-action/Dockerfile as an image with a random tag, runs the image with the supplied arguments and stops the container. On self-hosted runner a new image with the random tag remains for every use of the action. In most cases this is just a ne tag for the same image and does not use up disk space but I still end up with hundreds of tags for the same image.

Found this on stackoverflow (and added a link to my github action docs) on setting the user: stackoverflow.com/questions/58955666/how-to-set-the-docker-user-in-github-actions

This doesn't work, as the yq image does not contain git nor any other of the expected runner tools. So there is no way to use actions/checkout inside the yq image. The linked container usage is mainly there for running service containers.

[mikefarah]

I was referring to this bit in stackoverflow, where they specify the user in the container:

    container:
      image: owasp/zap2docker-stable
      options: --user root

Huh I had assumed github would build that image on release and put it in a docker repo. Hmm I could put the file in there if it helps - but I'd still need the github action docker file to set the USER and entrypoint commands right? I wonder if there's a better way where I can publish a docker image and just reference that in the action.yml so it doesn't need to build everytime

[mikefarah]

Oh wait is that what you mean over here:

runs:
  using: 'docker'
  image: 'mikefarah/yq:4'
  entrypoint: '/github/entrypoint.sh' 
  args:
    - ${{ inputs.cmd }}

like that goes in the action.yml?

[kuhnroyal]

Oh wait is that what you mean over here:

runs:
  using: 'docker'
  image: 'mikefarah/yq:4'
  entrypoint: '/github/entrypoint.sh' 
  args:
    - ${{ inputs.cmd }}

like that goes in the action.yml?

Yea exactly! But then the image uses the wrong user ID and there is currently no way to override it.

[mikefarah]

Ok I think the best way is to create and publish a specific github action image, that both has the entrypoint.sh and sets the user ID. Then have the action.yml reference that prebuilt image.

Just need time to update the pipeline...

[kuhnroyal]

Maybe the same image name, just a different tag?

[mikefarah]

Could do that - the only issue being you wouldn't be able to use dockers special latest syntax to always use the latest image...

[kuhnroyal]

Well yea, but that is not really needed for github actions. It is not currently being used in the action either. Could be a simple as just tagging the image with *-github:

runs:
  using: 'docker'
  image: 'mikefarah/yq:4-github'

[mikefarah]

Oh you are right, the current action just gets mikefarah/yq:4. Ok I think that makes sense. Now it just needs to be done .

[mikefarah]

Sorry this took a while - but I've just made the update, the action now uses a published docker image.

image: 'docker://mikefarah/yq:4-githubaction'

Note that the action has been updated as per github's deprecation notice: https://github.com/mikefarah/yq/issues/1382

If needed, you should be able to access the older github action still by:

uses: mikefarah/[email protected]

[kuhnroyal]

Awesome thanks! I will test this when I am back from vacation.

[kuhnroyal]

Thanks for the changes!