project-zot
[Bug]: The /tmp filesystem is filling up with Trivy directories
zot version
v2.1.11
Describe the bug
Hello,
Since updating from v2.1.7 to v2.1.11, the file system under /tmp/ has been filling up with 'trivy-xxxxxx'.
To reproduce
Updating from v2.1.7 to v2.1.11
- Configuration
{
"storage": {
"rootDirectory": "/srv/zot",
"GC": true,
"gcDelay": "1h",
"gcInterval": "168h",
"dedupe": true,
"retention": {
"dryRun": false,
"delay": "24h",
"policies": [{
"repositories": ["**"],
"deleteReferrers": true,
"deleteUntagged": true,
"keepTags": [{
"pulledWithin": "720h",
"pushedWithin": "720h"
}]
}]
}
},
"http": {
"address": "0.0.0.0",
"port": "5000",
"externalUrl": "https://oci.dev.local",
"compat": ["docker2s2"],
"tls": {
"cert": "/etc/zot/fullchain.pem",
"key": "/etc/zot/privkey.pem"
},
"auth": {
"openid": {
"providers": {
"oidc": {
"credentialsFile": "/etc/zot/oidc-credentials.json",
"issuer": "https://keycloak.local/realms/sub",
"name": "SSO",
"keypath": "",
"scopes": ["openid"]
}
}
}
},
"accessControl": {
"metrics":{
"users": ["admin@local"]
},
"repositories": {
"**": {
"policies": [
{
"groups": ["S_admin"],
"actions": ["read", "create", "update", "delete"]
}
],
"anonymousPolicy": ["read"],
"defaultPolicy": ["create", "read"]
}
}
}
},
"log": {
"level": "warn"
},
"extensions": {
"metrics": {
"enable": true,
"prometheus": {
"path": "/metrics"
}
},
"scrub": {
"enable": true,
"interval": "24h"
},
"search": {
"enable": true,
"cve": {
"updateInterval": "48h"
}
},
"ui": {
"enable": true
},
"sync": {
"credentialsFile": "/etc/zot/sync-auth-filepath.json",
"registries": [
{
"urls": ["https://ghcr.io"],
"onDemand": true
},
{
"urls": ["https://docker.gitea.com"],
"onDemand": true
}
]
}
}
}
- Client tool used
- Seen error
:~]# du -sh /tmp
59G /tmp
:~]# ll /tmp/
total 2780
drwxrwxrwt 17 root root 4096 Nov 28 08:19 ./
drwxr-xr-x 21 root root 4096 Nov 24 22:48 ../
....
drwxr-xr-x 3 zot zot 4096 Nov 27 15:25 trivy-455860/
drwxr-xr-x 2 zot zot 2777088 Nov 28 08:09 trivy-456294/
....
Expected behavior
No response
Screenshots
No response
Additional context
No response
Hi @zachelnet, wasn't this also the case before the upgrade?
Note those folders are created by the Trivy library we import for CVE scanning, we don't manage them directly. Do they contain only files such as /tmp/trivy-3359096/analyzer-file-286123778?
Hi @zachelnet, wasn't this also the case before the upgrade?
Note those folders are created by the Trivy library we import for CVE scanning, we don't manage them directly. Do they contain only files such as
/tmp/trivy-3359096/analyzer-file-286123778?
For the upgrade, the folder wasn't on /tmp
@andaaron on v2.1.7 is the trivy database on rootDirectory/_trivy/
~]# ll /srv/zot/_trivy/
total 20
drwxr-xr-x 5 zot zot 4096 Nov 28 21:47 ./
drwxr-xr-x 75 zot zot 4096 Nov 28 21:46 ../
drwxr-xr-x 2 zot zot 4096 Nov 28 21:47 db/
drwx------ 2 zot zot 4096 Aug 29 14:34 fanal/
drwxr-xr-x 2 zot zot 4096 Nov 27 15:31 java-db/
~]# ll /srv/zot/_trivy/db/
total 863492
drwxr-xr-x 2 zot zot 4096 Nov 28 21:47 ./
drwxr-xr-x 5 zot zot 4096 Nov 28 21:47 ../
-rw-r--r-- 1 zot zot 152 Nov 28 21:47 metadata.json
-rw-r--r-- 1 zot zot 884199424 Nov 28 19:23 trivy.db
after v2.1.11 is changing on /tmp/trivy-xxxxxx.
It's possible to change this?
Nothing has changed AFAIK. Trivy has been downloading the content under /tmp/ and moving it to /srv/zot/_trivy for as far as I can remember. I mentioned this in one of my comments: https://github.com/project-zot/project-zot.github.io/issues/138#issuecomment-1843112929, which will need to get into the documentation.
The analyzer files may be new, I don't remember seeing them before. I will take a look to see if they added a new customization to fix this. If not, maybe setting a different path for the TMPDIR environment variable for the zot process may help.
https://github.com/aquasecurity/trivy/discussions/9457#discussioncomment-14351910
It seems like they made this change https://github.com/aquasecurity/trivy/commit/8f5b56005a4e8752976524750089dc9ea2c91e40 in Trivy 0.65.0 which we picked in https://github.com/project-zot/zot/pull/3292.
Seems like they have new apis for creating and cleaning up the temporary folders, and the Cleanup is not explicitly called from our code, causing the "temporary" files to keep piling up under the default trivy directory (I see they have a 77 MB tarball with the database under /tmp/trivy-84376/oci-download-2700474122/db.tar.gz). Calling that Cleanup should be the correct fix in our case.
Hi @zachelnet, I have merged https://github.com/project-zot/zot/pull/3618 to main. From now on, the temporary files should be deleted as soon as they are no longer needed. Would you be willing to try it out?
The already existing /tmp/trivy should be cleaned up manually. They are related to previous zot executions, and are no longer needed.
All the important data we need to keep is already under /srv/zot/_trivy/, this has been the case in both v2.1.7 to v2.1.11.
@zachelnet pls verify this PR (merged) is working for your use case. https://github.com/project-zot/zot/pull/3618
If so, pls close this issue.
