crowdsecurity
Bug/Downloading collections/file creation after mounting volume issue
Describe the bug After I start the application in Docker, it throws errors that is unable to read files (if mounted as - $DOCKER/CrowdSec:/etc/crowdsec:rw) and restarts. To solve it, I needed to cp whole /etc/crowdsec container from inside docker and paste it to my volume, then the app starts. Seems like the container itself could not create everything. As I am using Docker on Synology, I understand I need to manually create folders, but even after doing so (no files) it still seems the app has issues creating own files. For example, linuxserver/nginx container has no issues at all creating folders and directories upon starting on mounting whole directory (- $DOCKER/Nginx:/config:rw). But the linuxserver's container uses two additional environments that are PUID and PGID that I set to my docker user that has access to the directory. If I am not wrong, Crowdsec seems to have GID but that did not work well to me. Maybe introducing the variables as linuxserver container would help this specific issue on my system? For now I can just cp whole directory but I am not sure how it would handle downloading collections..
And to the second issue once I copied the directory, it seems that I am unable to download collections. Log attached - CrowdSec.zip. All files are default from crowdsec container but config.yaml and acquis.yaml that configs I attach here: CrowdSecConfig.zip
To Reproduce Steps to reproduce the behavior:
- Use Docker on Synology (I am guessing, as it has a bit different permissions)
- Setup and start container
- Check logs
- See error
Expected behavior I would expect it to work and download collections and create default files on its run automatically as it makes me either cp directory from within container manually or mount specific config files I edit. Which in theory also would mean I can't keep collections on my mounted volume as it would have issues creating them (would probably work on volume inside docker, or without mouting at all).
Screenshots Screenshot of logs, that should be also attached above - http://prntscr.com/26v077j
Technical Information (please complete the following information):
- OS: Synology DSM 7
Docker Version: (newest on Synology branch) Client: Version: 20.10.3 API version: 1.41 Go version: go1.15.13 Git commit: b455053 Built: Thu Aug 19 07:13:24 2021 OS/Arch: linux/amd64 Context: default Experimental: true
Server: Engine: Version: 20.10.3 API version: 1.41 (minimum version 1.12) Go version: go1.15.13 Git commit: a3bc36f Built: Thu Aug 19 07:11:25 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.4.3 GitCommit: ea3508454ff2268c32720eb4d2fc9816d6f75f88 runc: Version: v1.0.0-rc93 GitCommit: 31cc25f16f5eba4d0f53e35374532873744f4b31 docker-init: Version: 0.19.0 GitCommit: ed96d00
docker-compose version 1.28.5, build 324b023a docker-py version: 4.4.4 CPython version: 3.7.10 OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
Additional context
Docker compose:
crowdsec: container_name: CrowdSec hostname: crowdsec image: crowdsecurity/crowdsec:latest restart: unless-stopped privileged: false networks: Proxy: # ports: # - 6060:6060/tcp # - 8080:8080/tcp security_opt: - no-new-privileges:true volumes: - $DOCKER/CrowdSec:/etc/crowdsec:rw - $DOCKER/CrowdSec/Data:/var/lib/crowdsec/data:rw #- $DOCKER/CrowdSec/acquis.yaml:/etc/crowdsec/acquis.yaml:rw #- $DOCKER/CrowdSec/config.yaml:/etc/crowdsec/config.yaml:rw - $DIR_LOGS:/var/log:ro environment: #https://hub.crowdsec.net/browse - GID=$PGID_DOCKERGROUP - COLLECTIONS="crowdsecurity/base-http-scenarios" - PARSERS="crowdsecurity/traefik-logs crowdsecurity/nginx-logs" - CONFIG_FILE=/etc/crowdsec/config.yaml - TEST_MODE=false - TZ=$TZ - CUSTOM_HOSTNAME=crowdsec ## Custom Variables - DB_TYPE=mysql - DB_USER=$MYSQL_CROWDSEC_ACCOUNT - DB_PASSWORD=$MYSQL_CROWDSEC_PASSWORD - DB_NAME=$MYSQL_CROWDSEC_DATABASE - DB_HOST=mariadb - DB_PORT=3306
Seems docker compose did not paste like I expected, here is more readable screenshot: http://prntscr.com/26v0cm8
Mysql connection works, app creates its entries. No bouncers are installed yet. Using the exactly same config from under here: https://github.com/crowdsecurity/crowdsec/blob/master/docker/config.yaml gives the same issue.
Hello @Maypul,
This supposed to be fixed by https://github.com/crowdsecurity/crowdsec/pull/1216 !
However, there was no release since the merge. Either I let you build/test, otherwise a new image is going to hopefully land soon :+1:
@buixor thanks for letting me know, must have missed it. I will wait for new release then. Though, do you know if it will help with the app is not downloading collections? Every other container seems to be connecting to the internet just fine. It is somehow related to the image not being able to mount thing correctly even after using whole copied directory?
I have downloaded the new :latest release from today, seems the issue is still here:
- with mounted config and acquis yamls: https://prnt.sc/jb8JyO7etx8g
- with whole dirs mounted: https://prnt.sc/hCAXXEbox2YP
- with no dirs mounted: https://prnt.sc/UtU0umRsxxLH (still unable to download collections) Also how would I go about registering with online api? There is an online credentials file but I could not figure out what to put in here.
@Maypul Actually the new release is supposed to fix the mount being empty issue by copying configuration files into the mount as part of the startup script. However this is done only if in the mount there's no config.yaml and local_api_credentials.yaml files, to avoid overwriting any configuration. So, if passing to the new release your local folder wasn't empty, this will not work.
Starting as new install on DSM 6 (don't have a DSM 7 to test but I imagine is rather similar), I just started the container with something like /volume1/docker/crowdsec:/etc/crowdsec:rw and ports mapping + COLLECTION environment variable. I've been able to install a new collection then and crowdsec works correctly, initially empty /volume1/docker/crowdsec has been populated with config files at first start.
In your case I'm wondering about the permissions on your $DOCKER folder ?
