nx-tools
nx-tools copied to clipboard
gperdomor
[@nx-tools/docker] Passing metadata options through environment variables
Hi,
Thanks for this fantastic set of tools. We need to make use of metadata options, and these are distinct in our CI setup, depending on the branch the user is on etcetera. So they need to be customizable in our GitHub Action.
I was able to provide various configuration options via INPUT_*, but I cannot figure out how to make it work for metadata options.
For example, I've tried INPUT_METADATA_TAGS=... but this does not get picked up.
Is there anything I am missing? Is the answer perhaps in the undocumented @nx-tools/docker-metadata package in this repo?
Any feedback/help is appreciated. Thanks!
@arielweinberger INPUT_TAGS should works... can you share the repo or the project.json config please
Hey @gperdomor thanks for taking a look. I am aware that INPUT_TAGS works, but I need the metadata variables, these ones:
I assumed env variables such as INPUT_METADATA_TAGS, INPUT_METADATA_FLAVOR etc would work but they didn't. Unfortunately I don't have the code for that anymore as I have found another solution.
What I ended up doing is using a separate GitHub workflow job step to output the tags based on metadata, and import them into my dockerization executor.
# https://github.com/docker/metadata-action
- name: Docker Metadata
uses: docker/metadata-action@v4
id: meta
with:
images: "407256539111.dkr.ecr.us-east-1.amazonaws.com/${{ matrix.project }}-nx-test"
tags: |
type=sha,prefix=sha-
type=ref,event=branch
type=ref,event=pr
type=ref,event=tag
- name: Dockerize
run: npx nx run-many --target docker:build --projects=${{ matrix.project }}
env:
# https://www.npmjs.com/package/@nx-tools/nx-docker
INPUT_TAGS: ${{ steps.meta.outputs.tags }}
INPUT_PUSH: true
:o ok ok... Well in first place you not need to use the docker/metadata-action, @nx-tools/docker-metadata handles that for you, the only requirement is to setup the metadata.images directly in your project.json
{
"name": "api",
"$schema": "../../node_modules/nx/schemas/project-schema.json",
"sourceRoot": "apps/api/src",
"projectType": "application",
"targets": {
"build": { ... },
"serve": { ... },
"lint": { ... },
"test": { ... },
"docker": {
"executor": "@nx-tools/nx-docker:build",
"options": {
"metadata": {
"images": ["arielweinberger/api"], <-- THIS IS REQUIRED for the automatic metadata flow, don't worry about the value, can be overriden with INPUT_IMAGES or with INPUT_API_IMAGES (because the name of the project is api)
"tags": [
"type=sha,prefix=sha-",
"type=ref,event=branch",
"type=ref,event=pr",
"type=ref,event=tag"
]
}
}
}
}
}
All accepted env variables for metadata are: INPUT_BAKE_TARGET, INPUT_GITHUB_TOKEN, INPUT_SEP_LABELS, INPUT_SEP_TAGS, INPUT_FLAVOR, INPUT_IMAGES, INPUT_LABELS, INPUT_TAGS, and the "prefixed" alternatives with the name of the project like INPUT_(PROJECT)_IMAGES, etc...
With the prefixed alternatives you can build everything without a matrix:
- name: Dockerize
run: npx nx affected --base=$NX_BASE --head=$NX_HEAD --target=docker --parallel=2
env:
// Common vars
INPUT_PUSH: true
// Specific for each project
INPUT_API_TAGS: 407256539111.dkr.ecr.us-east-1.amazonaws.com/api-nx-test
INPUT_WEB_TAGS: 407256539111.dkr.ecr.us-east-1.amazonaws.com/web-nx-test
INPUT_GATEWAY_TAGS: 407256539111.dkr.ecr.us-east-1.amazonaws.com/gateway-nx-test
btw for those interested you can always use the CLI to override those values:
npx nx run my-app:container -- --metadata.images={myapp,image}
however, this does not work for only a single image and not for an empty image name. The schema must be array. which I wasn't able to figure out how to do a "singled value array" :/
@mrsufgi @gperdomor Thanks. We'll look into that. For the time being, we are still using the Docker Metadata action on GitHub Actions, we found it to be robust and reliable.
Honestly I think its there should be a better way, the fact that you can't mix and match container metadata with setting the baseimage via either CLI or ENV is counterintuitive tbh.
It mostly make the push hard to use. In my project I don't know the Docker registry URL, so I don't know what to put in "images" beforehand. if I put a generic "images" name. I can't do anything robust enough.
say I have 20 projects that I build and I do run-affected, I would have wanted to be able to set the "registry", the "repository" and let the container metadata thingy add the tag ("latest", "type=ref,event=pr").
The way I will be solving this is by ENV:
INPUT_API_TAGS: 407256539111.dkr.ecr.us-east-1.amazonaws.com/api-nx-test:commit-sha256, 407256539111.dkr.ecr.us-east-1.amazonaws.com/api-nx-test:latest
...
...
...
@mrsufgi @gperdomor Thanks. We'll look into that. For the time being, we are still using the Docker Metadata action on GitHub Actions, we found it to be robust and reliable.
That functionality is builded inside the plugin, give a try :D
@mrsufgi for what you want, just configure your target like this:
"container": {
"executor": "@nx-tools/nx-container:build",
"options": {
"metadata": {
"images": ["407256539111.dkr.ecr.us-east-1.amazonaws.com/api-nx-test"],
"tags": [
"type=schedule",
"type=ref,event=branch",
"type=ref,event=tag",
"type=ref,event=pr",
"type=semver,pattern={{version}}",
"type=semver,pattern={{major}}.{{minor}}",
"type=semver,pattern={{major}}",
"type=sha,prefix=sha-"
]
}
}
}
the commit-sha256, latests, branch-name, tag-version (if exists), in the end of the tag will be handled automatically...
Also you can put "images": ["api-nx-test"] for build in local and use the INPUT_TAGS or INPUT_{APPNAME}_TAGS to set the proper name in CI
That's what I meant, since I can only use "images": ["api-nx-test"] (I don't know/I have different ECR for different stages)
I set INPUT_{APPNAME}_TAGS (with my actual ECR url), then I lose all "automatically handled" tags: api-nx-test:sha..., api-nx-test:branch-name
I wish I could just set the base images and still get the container metadata.
@gperdomor, is it possible to have a documentation on @nx-tools/docker-metadata package? It's bit complicated to correctly set it up to suit our needs...
:o ok ok... Well in first place you not need to use the
docker/metadata-action,@nx-tools/docker-metadatahandles that for you, the only requirement is to setup the metadata.images directly in your project.json{ "name": "api", "$schema": "../../node_modules/nx/schemas/project-schema.json", "sourceRoot": "apps/api/src", "projectType": "application", "targets": { "build": { ... }, "serve": { ... }, "lint": { ... }, "test": { ... }, "docker": { "executor": "@nx-tools/nx-docker:build", "options": { "metadata": { "images": ["arielweinberger/api"], <-- THIS IS REQUIRED for the automatic metadata flow, don't worry about the value, can be overriden with INPUT_IMAGES or with INPUT_API_IMAGES (because the name of the project is api) "tags": [ "type=sha,prefix=sha-", "type=ref,event=branch", "type=ref,event=pr", "type=ref,event=tag" ] } } } } }All accepted env variables for metadata are: INPUT_BAKE_TARGET, INPUT_GITHUB_TOKEN, INPUT_SEP_LABELS, INPUT_SEP_TAGS, INPUT_FLAVOR, INPUT_IMAGES, INPUT_LABELS, INPUT_TAGS, and the "prefixed" alternatives with the name of the project like INPUT_(PROJECT)_IMAGES, etc...
With the prefixed alternatives you can build everything without a matrix:
- name: Dockerize run: npx nx affected --base=$NX_BASE --head=$NX_HEAD --target=docker --parallel=2 env: // Common vars INPUT_PUSH: true // Specific for each project INPUT_API_TAGS: 407256539111.dkr.ecr.us-east-1.amazonaws.com/api-nx-test INPUT_WEB_TAGS: 407256539111.dkr.ecr.us-east-1.amazonaws.com/web-nx-test INPUT_GATEWAY_TAGS: 407256539111.dkr.ecr.us-east-1.amazonaws.com/gateway-nx-test
Hello @gperdomor
I successfully tested the override of image name in my pipeline with version 5.0.3. But it's not working anymore since we upgrade to version 6.0.0
Could you please update the documentation with how to override the image name in an NX multi project please ?
Thanks
@w8ze-devel hi buddy... try with 6.0.1 because initial 6.0.0 had a problem reading some ends and was reported here https://github.com/gperdomor/nx-tools/issues/1045...
Let me know if 6.0.1 works for you 🙏🏻
I have the same problem as @mrsufgi
The crux of the issue is that my REGISTRY_NAME is only known at build time, and the set of applications built by nx affected ... is only known (by nx) at build time.
The ideal solution in my case would be passing an "INPUT_REGISTRY" environment variable which the nx-container plugin would combine with the hard-coded image value from the project.json. The project.json might also support a registry setting but that would still need to accept build-time configuration for my case.
Indeed, the docker build documentation even refers to these two values independently in this exact context:
docker buildx build --tag <registry>/<image> --push .
So this seems like a natural extension of the plugin to support a registry option that is configurable at build time. I saw you mention enumerating INPUT_PROJECT_TAGS environment variables for all of the projects but that couples the GitHub workflow yaml to the specific project names in the monorepo, which feels like an anti-pattern.
