trivy
trivy copied to clipboard
aquasecurity
AWS ECR registry authentication only works in the same/default region as caller
Description
Trivy is deployed within an AWS EKS Kubernetes pod. The pod has an AWS role associated to read ECR images. Trivy is executed to scan:
- Private AWS ECR images
- Public AWS ECR images in the same region
- Public AWS ECR images in other regions
What did you expect to happen?
Trivy is able to scan these three image types. Authentication is handled transparently by Trivy (for all these types). On failure, an available docker-credential-ecr-login helper is used/tried, too.
What happened instead?
Trivy is unable to scan (public) ECR images in other AWS regions (except the region of the EKS cluster).
The automatic AWS authentication handler injects ECR authentication tokens from the current (default) region, not the region of the ECR image itself.
Having the docker-credential-ecr-login helper available and configured does not mitigate the authentication issue (the helper is never used by Trivy; although it would return valid credentials). There is no known configuration option to force Trivy to fall back to using the docker-credential-ecr-login helper.
In our case, we start Trivy in an EKS cluster in us-east-1. It is able to scan private and public ECR images in us-east-1. Public images in other regions like eu-central-1 fail (I could not yet test it with private ECR images).
Output of run with -debug:
/ # trivy --debug image --skip-update --severity CRITICAL 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/coredns:v1.7.0-eksbuild.1
2021-05-27T14:20:50.859Z DEBUG Severities: CRITICAL
2021-05-27T14:20:50.860Z DEBUG cache dir: /root/.cache/trivy
2021-05-27T14:20:50.861Z DEBUG DB Schema: 1, Type: 1, UpdatedAt: 2021-05-27 12:13:39.579564022 +0000 UTC, NextUpdate: 2021-05-28 00:13:39.579563622 +0000 UTC, DownloadedAt: 2021-05-27 13:37:54.854680347 +0000 UTC
2021-05-27T14:20:50.861Z DEBUG Vulnerability type: [os library]
2021-05-27T14:20:51.204Z DEBUG Image ID: sha256:e3adaca0b74ac9810f91d8ae39d80681772f4a1338201e498fb4abdd0b80e5f3
2021-05-27T14:20:51.204Z DEBUG Diff IDs: [sha256:32e8e94d13e789ff20ec4ea815f971bbcc3ce52955c6ac8b7ce25546ad978024 sha256:b0bc0874083cf0b4db645481dcd8d4b5fa0ab02295676c19d8a4f3960eaf711e]
2021-05-27T14:20:51.205Z DEBUG OS is not detected and vulnerabilities in OS packages are not detected.
2021-05-27T14:20:51.205Z INFO Detected OS: unknown
2021-05-27T14:20:51.205Z INFO Number of PL dependency files: 1
2021-05-27T14:20:51.205Z INFO Detecting gobinary vulnerabilities...
2021-05-27T14:20:51.206Z DEBUG Detecting library vulnerabilities, type: gobinary, path: coredns
coredns
=======
Total: 0 (CRITICAL: 0)
/ # trivy --debug image --skip-update --severity CRITICAL 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/coredns:v1.7.0-eksbuild.1
2021-05-27T14:21:04.191Z DEBUG Severities: CRITICAL
2021-05-27T14:21:04.192Z DEBUG cache dir: /root/.cache/trivy
2021-05-27T14:21:04.192Z DEBUG DB Schema: 1, Type: 1, UpdatedAt: 2021-05-27 12:13:39.579564022 +0000 UTC, NextUpdate: 2021-05-28 00:13:39.579563622 +0000 UTC, DownloadedAt: 2021-05-27 13:37:54.854680347 +0000 UTC
2021-05-27T14:21:04.192Z DEBUG Vulnerability type: [os library]
2021-05-27T14:21:04.903Z FATAL scan error:
github.com/aquasecurity/trivy/pkg/commands/artifact.runWithTimeout
/home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:67
- unable to initialize a scanner:
github.com/aquasecurity/trivy/pkg/commands/artifact.scan
/home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:157
- unable to initialize a docker scanner:
github.com/aquasecurity/trivy/pkg/commands/artifact.dockerScanner
/home/runner/work/trivy/trivy/pkg/commands/artifact/image.go:29
- 3 errors occurred:
* unable to inspect the image (602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/coredns:v1.7.0-eksbuild.1): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
* unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory
* GET https://602401143452.dkr.ecr.eu-central-1.amazonaws.com/v2/eks/coredns/manifests/v1.7.0-eksbuild.1: DENIED: Your Authorization Token is invalid.
Output of trivy -v:
Version: 0.18.3
Issue can be reproduced with at least 0.16.0, too.
Additional details (base image name, container registry info...):
Produce same error messages with built-in tools
The error message Your Authorization Token is invalid. can be reproduced by providing an AWS authorization token from a different region:
/ # AWS_TOKEN_EU_CENTRAL_1=$(aws --region=eu-central-1 ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken')
/ # curl -i -H "Authorization: Basic $AWS_TOKEN_EU_CENTRAL_1" https://602401143452.dkr.ecr.eu-central-1.amazonaws.com/v2/eks/coredns/manifests/v1.7.0-eksbuild.1
HTTP/1.1 200 OK
Content-Type: application/vnd.docker.distribution.manifest.list.v2+json
Docker-Distribution-Api-Version: registry/2.0
Date: Thu, 27 May 2021 15:12:48 GMT
Content-Length: 741
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 739,
"digest": "sha256:0e681df097589fa1cdd6c09bf140d9dd9ade86f2cadca06fa6a5008ac7da1cd2",
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 739,
"digest": "sha256:3174688c1f4bcc94c786c22e2b3c7cc8e171b1b3997f8ddc777152766618ef6e",
"platform": {
"architecture": "arm64",
"os": "linux"
}
}
]
/ # AWS_TOKEN_US_EAST_1=$(aws --region=us-east-1 ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken')
/ # curl -i -H "Authorization: Basic $AWS_TOKEN_US_EAST_1" https://602401143452.dkr.ecr.eu-central-1.amazonaws.com/v2/eks/coredns/manifests/v1.7.0-eksbuild.1
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Date: Thu, 27 May 2021 15:12:59 GMT
Content-Length: 80
{"errors":[{"code":"DENIED","message":"Your Authorization Token is invalid."}]}
Authentication via docker-credential-ecr-login would work
Previously, the pod assumed its role via an older kube2iam version (0.10.x). Apparently, the AWS API imitated by kube2iam was not sufficient to convince Trivy to be running in an AWS context. It did not handle the authentication itself. The available docker-credential-ecr-login helper was successfully used for all image types:
/ # docker-credential-ecr-login list
{}
/ # trivy --debug image --skip-update --severity CRITICAL 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/coredns:v1.7.0-eksbuild.1
2021-05-27T15:27:35.912Z DEBUG Severities: CRITICAL
2021-05-27T15:27:35.913Z DEBUG cache dir: /root/.cache/trivy
2021-05-27T15:27:35.913Z DEBUG DB Schema: 1, Type: 1, UpdatedAt: 2021-05-27 12:13:39.579564022 +0000 UTC, NextUpdate: 2021-05-28 00:13:39.579563622 +0000 UTC, DownloadedAt: 2021-05-27 13:37:50.414451 +0000 UTC
2021-05-27T15:27:35.913Z DEBUG Vulnerability type: [os library]
2021-05-27T15:27:36.381Z DEBUG Image ID: sha256:e3adaca0b74ac9810f91d8ae39d80681772f4a1338201e498fb4abdd0b80e5f3
2021-05-27T15:27:36.381Z DEBUG Diff IDs: [sha256:32e8e94d13e789ff20ec4ea815f971bbcc3ce52955c6ac8b7ce25546ad978024 sha256:b0bc0874083cf0b4db645481dcd8d4b5fa0ab02295676c19d8a4f3960eaf711e]
2021-05-27T15:27:36.382Z DEBUG OS is not detected and vulnerabilities in OS packages are not detected.
2021-05-27T15:27:36.382Z INFO Detected OS: unknown
2021-05-27T15:27:36.382Z INFO Number of PL dependency files: 1
2021-05-27T15:27:36.382Z INFO Detecting gobinary vulnerabilities...
2021-05-27T15:27:36.382Z DEBUG Detecting library vulnerabilities, type: gobinary, path: coredns
coredns
=======
Total: 0 (CRITICAL: 0)
/ # docker-credential-ecr-login list
{"https://602401143452.dkr.ecr.us-east-1.amazonaws.com":"AWS"}
/ # trivy --debug image --skip-update --severity CRITICAL 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/coredns:v1.7.0-eksbuild.1
2021-05-27T15:27:04.407Z DEBUG Severities: CRITICAL
2021-05-27T15:27:04.408Z DEBUG cache dir: /root/.cache/trivy
2021-05-27T15:27:04.408Z DEBUG DB Schema: 1, Type: 1, UpdatedAt: 2021-05-27 12:13:39.579564022 +0000 UTC, NextUpdate: 2021-05-28 00:13:39.579563622 +0000 UTC, DownloadedAt: 2021-05-27 13:37:50.414451 +0000 UTC
2021-05-27T15:27:04.408Z DEBUG Vulnerability type: [os library]
2021-05-27T15:27:06.164Z DEBUG Image ID: sha256:e3adaca0b74ac9810f91d8ae39d80681772f4a1338201e498fb4abdd0b80e5f3
2021-05-27T15:27:06.164Z DEBUG Diff IDs: [sha256:32e8e94d13e789ff20ec4ea815f971bbcc3ce52955c6ac8b7ce25546ad978024 sha256:b0bc0874083cf0b4db645481dcd8d4b5fa0ab02295676c19d8a4f3960eaf711e]
2021-05-27T15:27:06.165Z DEBUG OS is not detected and vulnerabilities in OS packages are not detected.
2021-05-27T15:27:06.165Z INFO Detected OS: unknown
2021-05-27T15:27:06.165Z INFO Number of PL dependency files: 1
2021-05-27T15:27:06.165Z INFO Detecting gobinary vulnerabilities...
2021-05-27T15:27:06.165Z DEBUG Detecting library vulnerabilities, type: gobinary, path: coredns
coredns
=======
Total: 0 (CRITICAL: 0)
/ # docker-credential-ecr-login list
{"https://602401143452.dkr.ecr.eu-central-1.amazonaws.com":"AWS","https://602401143452.dkr.ecr.us-east-1.amazonaws.com":"AWS"}
yes, @knqyf263 I opened the same issue: https://github.com/aquasecurity/trivy/issues/1034 and realized someone had already opened this one, but I will go close it now, thanks for the tips
any possibility that you provide some leads on how trivy reads ~/.aws/config? I would like to switch a few regions back and fourth, would be nice to understand that rather than going to switch [default] region=<region> every time, if possible
thanks!
@knqyf263 Specifying a matching region to the image location via AWS_REGION / AWS_DEFAULT_REGION resolves the issue.
I discovered that EKS clusters have a pod-identiy-webhook configured to inject needed environment variables to assume roles via OIDC / web-identity. This also injects AWS_REGION and AWS_DEFAULT_REGION based on the cluster location if they are not specified.
If both variables are unset or blank, Trivy correctly auto-selects the region based on the image location.
Creating the pod with explicit black values like
env:
- name: AWS_DEFAULT_REGION
value: ""
- name: AWS_REGION
value: ""
prevents the webhook from inject region environment values: Trivy's auto-detection works as expected. I wasn't able to resolve the issue by creating a ~/.aws/config file.
But this still fells like some kind of workaround - running Trivy in an EKS cluster to scan ECR images appears to me like a common use-case. The documentation on the ECR integration page reads like this should work out-of-the-box.
Is it possible to override the region directly in Trivy? Otherwise, it feels like this pitfall is worth to document.
This issue is stale because it has been labeled with inactivity.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
I have a workaround in-place (blank AWS_REGION, AWS_DEFAULT_REGION) but would prefer a proper solution in Trivy if reasonable.
This issue is stale because it has been labeled with inactivity.
Situation hasn't changed on my site, but I would prefer a mentation statement regarding option to improve the situation of future users.
This issue is stale because it has been labeled with inactivity.
I ran across this issue in version 0.28.0 (couldn't use 0.30.0 due to https://github.com/aquasecurity/trivy/issues/2349) trying to troubleshoot both trivy image and trivy k8s plugins scanning images stored in private ECR repository.
This is just running trivy on an EC2 instance with access to the EKS cluster API, not running it as a pod or operator. The instance has a profile with associated role to allow access to EKS and ECR.
Using trivy image without AWS_REGION set results in the same errors mentioned in the original report.
Using trivy k8s without AWS_REGION set results in the scan completing without any error but many images are simply not scanned and there is no warning/error/output that it wasn't able to scan anything.
The workaround suggested by @msw-kialo to set AWS_REGION or AWS_DEFAULT_REGION as empty strings did not seem to work for me, I could not trigger the auto-selection of region.
However I was able to get it working by installing docker, starting docker daemon, adding the user to the docker group, and installing/enabling config for docker-credential-ecr-login.
It would be great if trivy k8s provided some output on images it was unable to scan because right now it gives the false impression that it didn't find any vulnerabilities.
This issue is stale because it has been labeled with inactivity.
While we have a workaround in place, a proper resolution would be great (I am reluctant to try fixing it myself as I am not familiar with the code base).
@msw-kialo We've added --aws-region in VM scanning. Do you think adding the same option to image scanning helps?
https://github.com/aquasecurity/trivy/pull/3284
@msw-kialo did you try
However I was able to get it working by installing docker, starting docker daemon, adding the user to the docker group, and installing/enabling config for docker-credential-ecr-login.
@sinner- I didn't try adding a docker daemon. It is IMO also a workaround. However, it requires much more maintenance attention and further permissions (at least if you run in a Kubernetes cluster). Unsetting a few environments variables is easier.
@knqyf263 Based on the PR, the --aws-region parameter is specific to VM scanning (we scan container images). However, the PR shows that it might be feasible to extract an aws-region from the image reference that is about to be scanned and use that region for authentication :thinking:
Based on the PR, the --aws-region parameter is specific to VM scanning (we scan container images).
Yes, I'm asking if you want the option for image scanning. The CLI flag can overwrite environmental variables injected by webhook.
Sorry (apparently I skipped over the second sentence).
Yes, such an option would be good to have:
- It would allow users to resolve such permission issues without digging into injected environment variables. In some cases it might be difficult to change that.
- The corresponding parameter description could help users understand such issues without internet searches; if there is a region parameter, it might be necessary to provide the correct value.
The optimal solution would auto-detect the region parameter based on the image registry URL. If it is an AWS managed registry (and only for those this parameter makes sense), the full registry URL will already contain the correct region: e.g.,
602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/coredns:v1.7.0-eksbuild.1 (us-east-1) and 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/coredns:v1.7.0-eksbuild.1 (eu-central-1).
With such autodetection in place, the CLI parameter might no longer be needed (providing a different value would almost always result in a permission issue). That is the difference to EBS / AMI references (they don't include the region).
OK, it seems like the region is always included. We can take it and fall back into a blank region when it cannot be detected. https://docs.aws.amazon.com/AmazonECR/latest/userguide/Repositories.html
It could be easy to do that. Here is the ECR logic. https://github.com/aquasecurity/trivy/blob/50fe43f14c655c5e7bca45f0a0e4bd92891487d9/pkg/fanal/image/registry/ecr/ecr.go
Apparently, this issue has been fixed in the meantime (either in trivy itself or the use aws libraries).
I tested trivy 0.48.1 without the added workaround (ensure trivy is called without AWS_REGION and AWS_DEFAULT_REGION): even with AWS_REGION and AWS_DEFAULT_REGION set to us-east-1, trivy was able to successfully scan images in other regions like eu-central-1.
This issue still exists with the latest version.
setting AWS_REGION and AWS_DEFAULT_REGION to "" fails for multiple regions
2024-02-26T18:56:03.738Z ERROR Error during vulnerabilities or misconfiguration scan: scan error: unable to initialize a scanner: unable to initialize an image scanner: 4 errors occurred:
* docker error: unable to inspect the image (127647282379.dkr.ecr.us-east-1.amazonaws.com/undistro-test-image:1.25.3): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
* containerd error: containerd socket not found: /run/containerd/containerd.sock
* podman error: unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory
* remote error: GET https://127647282379.dkr.ecr.us-east-1.amazonaws.com/v2/undistro-test-image/manifests/1.25.3: unexpected status code 401 Unauthorized: Not Authorized
2024-02-26T18:56:03.738Z ERROR Error during vulnerabilities or misconfiguration scan: scan error: unable to initialize a scanner: unable to initialize an image scanner: 4 errors occurred:
* docker error: unable to inspect the image (127647282379.dkr.ecr.sa-east-1.amazonaws.com/undistro-test-image:1.25.3): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
* containerd error: containerd socket not found: /run/containerd/containerd.sock
* podman error: unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory
* remote error: GET https://127647282379.dkr.ecr.sa-east-1.amazonaws.com/v2/undistro-test-image/manifests/1.25.3: unexpected status code 401 Unauthorized: Not Authorized
By default it uses the EKS region, and overriding it allows us to access an alternative region.
In order to scan images from multiple regions we need to modify the code, I have something working and will submit a PR shortly
