Docker mounting local api credentials results in no such file or directory for config.yaml

[ToshY]

What happened?

When testing #2121 with crowdsec docker image v1.5.3-rc4-debian, I've found that my container crashes when I try to mount local_api_credentials.yaml. The first initial fatal message shows the following:

crowdsec-crowdsec-1  | ++(docker_start.sh:50): cscli(): command cscli -c /etc/crowdsec/config.yaml config show-yaml
crowdsec-crowdsec-1  | ++(docker_start.sh:59): conf_get(): yq e .api.client.credentials_path
crowdsec-crowdsec-1  | time="13-08-2023 19:45:32" level=fatal msg="while reading yaml file: open /etc/crowdsec/config.yaml: no such file or directory"

Then it continues and shows another error:

crowdsec-crowdsec-1  | +(docker_start.sh:21): isfalse(): return 0
crowdsec-crowdsec-1  | +(docker_start.sh:194): yq -e '.login==strenv(CUSTOM_HOSTNAME)' null
crowdsec-crowdsec-1  | Error: open null: no such file or directory

Finally, it shows a last fatal message and the container dies:

crowdsec-crowdsec-1  | +(docker_start.sh:201): cscli machines add localhost --auto
crowdsec-crowdsec-1  | +(docker_start.sh:50): cscli(): command cscli -c /etc/crowdsec/config.yaml machines add localhost --auto
crowdsec-crowdsec-1  | time="13-08-2023 19:45:32" level=fatal msg="while reading yaml file: open /etc/crowdsec/config.yaml: no such file or directory"

Complete trace here:

docker compose logs -ft

Attaching to crowdsec-crowdsec-1, crowdsec-mysql-1
crowdsec-crowdsec-1  | + export 'PS4=+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
crowdsec-crowdsec-1  | + PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
crowdsec-crowdsec-1  | +(docker_start.sh:30): istrue ''
crowdsec-crowdsec-1  | +(docker_start.sh:11): istrue(): case "$(echo "$1" | tr '[:upper:]' '[:lower:]')" in
crowdsec-crowdsec-1  | ++(docker_start.sh:11): istrue(): echo ''
crowdsec-crowdsec-1  | ++(docker_start.sh:11): istrue(): tr '[:upper:]' '[:lower:]'
crowdsec-crowdsec-1  | +(docker_start.sh:13): istrue(): return 1
crowdsec-crowdsec-1  | +(docker_start.sh:36): export CONFIG_FILE=/etc/crowdsec/config.yaml
crowdsec-crowdsec-1  | +(docker_start.sh:36): CONFIG_FILE=/etc/crowdsec/config.yaml
crowdsec-crowdsec-1  | +(docker_start.sh:37): export CUSTOM_HOSTNAME=localhost
crowdsec-crowdsec-1  | +(docker_start.sh:37): CUSTOM_HOSTNAME=localhost
crowdsec-crowdsec-1  | +(docker_start.sh:146): '[' -n '' ']'
crowdsec-crowdsec-1  | +(docker_start.sh:146): '[' -n '' ']'
crowdsec-crowdsec-1  | +(docker_start.sh:157): for geodb in GeoLite2-ASN.mmdb GeoLite2-City.mmdb
crowdsec-crowdsec-1  | +(docker_start.sh:162): '[' '!' -e /var/lib/crowdsec/data/GeoLite2-ASN.mmdb ']'
crowdsec-crowdsec-1  | +(docker_start.sh:157): for geodb in GeoLite2-ASN.mmdb GeoLite2-City.mmdb
crowdsec-crowdsec-1  | +(docker_start.sh:162): '[' '!' -e /var/lib/crowdsec/data/GeoLite2-City.mmdb ']'
crowdsec-crowdsec-1  | +(docker_start.sh:169): '[' '!' -e /etc/crowdsec/local_api_credentials.yaml ']'
crowdsec-crowdsec-1  | +(docker_start.sh:182): istrue ''
crowdsec-crowdsec-1  | +(docker_start.sh:11): istrue(): case "$(echo "$1" | tr '[:upper:]' '[:lower:]')" in
crowdsec-crowdsec-1  | ++(docker_start.sh:11): istrue(): echo ''
crowdsec-crowdsec-1  | ++(docker_start.sh:11): istrue(): tr '[:upper:]' '[:lower:]'
crowdsec-crowdsec-1  | +(docker_start.sh:13): istrue(): return 1
crowdsec-crowdsec-1  | +(docker_start.sh:184): '[' -n '' ']'
crowdsec-crowdsec-1  | ++(docker_start.sh:188): conf_get .api.client.credentials_path
crowdsec-crowdsec-1  | ++(docker_start.sh:56): conf_get(): '[' 1 -ge 2 ']'
crowdsec-crowdsec-1  | ++(docker_start.sh:59): conf_get(): cscli config show-yaml
crowdsec-crowdsec-1  | ++(docker_start.sh:50): cscli(): command cscli -c /etc/crowdsec/config.yaml config show-yaml
crowdsec-crowdsec-1  | ++(docker_start.sh:59): conf_get(): yq e .api.client.credentials_path
crowdsec-crowdsec-1  | time="13-08-2023 19:45:32" level=fatal msg="while reading yaml file: open /etc/crowdsec/config.yaml: no such file or directory"
crowdsec-crowdsec-1  | +(docker_start.sh:188): lapi_credentials_path=null
crowdsec-crowdsec-1  | +(docker_start.sh:190): isfalse ''
crowdsec-crowdsec-1  | +(docker_start.sh:18): isfalse(): istrue ''
crowdsec-crowdsec-1  | +(docker_start.sh:11): istrue(): case "$(echo "$1" | tr '[:upper:]' '[:lower:]')" in
crowdsec-crowdsec-1  | ++(docker_start.sh:11): istrue(): tr '[:upper:]' '[:lower:]'
crowdsec-crowdsec-1  | ++(docker_start.sh:11): istrue(): echo ''
crowdsec-crowdsec-1  | +(docker_start.sh:13): istrue(): return 1
crowdsec-crowdsec-1  | +(docker_start.sh:21): isfalse(): return 0
crowdsec-crowdsec-1  | +(docker_start.sh:193): isfalse ''
crowdsec-crowdsec-1  | +(docker_start.sh:18): isfalse(): istrue ''
crowdsec-crowdsec-1  | +(docker_start.sh:11): istrue(): case "$(echo "$1" | tr '[:upper:]' '[:lower:]')" in
crowdsec-crowdsec-1  | ++(docker_start.sh:11): istrue(): echo ''
crowdsec-crowdsec-1  | ++(docker_start.sh:11): istrue(): tr '[:upper:]' '[:lower:]'
crowdsec-crowdsec-1  | +(docker_start.sh:13): istrue(): return 1
crowdsec-crowdsec-1  | +(docker_start.sh:21): isfalse(): return 0
crowdsec-crowdsec-1  | +(docker_start.sh:194): yq -e '.login==strenv(CUSTOM_HOSTNAME)' null
crowdsec-crowdsec-1  | Error: open null: no such file or directory
crowdsec-crowdsec-1  | +(docker_start.sh:197): echo 'Generate local agent credentials'
crowdsec-crowdsec-1  | +(docker_start.sh:200): cscli machines delete localhost
crowdsec-crowdsec-1  | Generate local agent credentials
crowdsec-crowdsec-1  | +(docker_start.sh:200): true
crowdsec-crowdsec-1  | +(docker_start.sh:201): cscli machines add localhost --auto
crowdsec-crowdsec-1  | +(docker_start.sh:50): cscli(): command cscli -c /etc/crowdsec/config.yaml machines add localhost --auto
crowdsec-crowdsec-1  | time="13-08-2023 19:45:32" level=fatal msg="while reading yaml file: open /etc/crowdsec/config.yaml: no such file or directory"

---

Seems that it looks for a config.yaml, but it can't find it. Not sure why.

What did you expect to happen?

Correctly mounting with local api credentials and starting up the container again.

How can we reproduce it (as minimally and precisely as possible)?

Mount local_api_credentials.yaml to /etc/crowdsec/local_api_credentials.yaml.

docker-compose.yml

version: '3.9'

x-restart-stopped: &restart-stopped
  restart: unless-stopped

x-restart-always: &restart-always
  restart: always

services:
  crowdsec:
    image: crowdsecurity/crowdsec:v1.5.3-rc4-debian
    depends_on:
      - mysql
    environment:
      DEBUG: 'TRUE'
      LEVEL_TRACE: 'TRUE'
      COLLECTIONS: ${CROWDSEC_COLLECTIONS}
      PARSERS: ${CROWDSEC_PARSERS}
      POSTOVERFLOWS: ${CROWDSEC_POSTOVERFLOWS}
      DATABASE_TYPE: ${MYSQL_HOST}
      DATABASE_USER: ${MYSQL_USER}
      DATABASE_USER_PASSWORD: ${MYSQL_PASSWORD}
      DATABASE_NAME: ${MYSQL_DATABASE}
      DATABASE_HOST: ${MYSQL_HOST}
      DATABASE_PORT: ${MYSQL_PORT}
      BOUNCER_KEY_TRAEFIK: ${CROWDSEC_BOUNCER_API_KEY}
    volumes:
      - ./data/crowdsec/config/local_api_credentials.yaml:/etc/crowdsec/local_api_credentials.yaml
      - ./data/crowdsec/config/online_api_credentials.yaml:/etc/crowdsec/online_api_credentials.yaml
      - ./data/crowdsec/config/acquis.d:/etc/crowdsec/acquis.d
      - ./data/crowdsec/config/config.yaml.local:/etc/crowdsec/config.yaml.local
      - ./data/crowdsec/config/profiles.yaml.local:/etc/crowdsec/profiles.yaml.local
      - ./data/crowdsec/notifications/discord.yaml:/etc/crowdsec/notifications/discord.yaml
      - ./data/crowdsec/parsers/s02-enrich/user-whitelist.yaml:/etc/crowdsec/parsers/s02-enrich/user-whitelist.yaml
      - ./data/crowdsec/postoverflows/s01-whitelist/fqdn-whitelists.yaml:/etc/crowdsec/postoverflows/s01-whitelist/fqdn-whitelists.yaml
      - crowdsec_datax:/var/lib/crowdsec/data
      - /var/log:/var/log:ro
      - proxy_logs:/log/traefik:ro
    ports:
      - "8282:8080"
    labels:
      traefik.enable: true
      traefik.http.routers.crowdsec.service: crowdsec
      traefik.http.services.crowdsec.loadbalancer.server.port: 8080
    networks:
      - crowdsec
      - proxy

  mysql:
    build:
      context: data/mysql
      dockerfile: Dockerfile
      args:
        MYSQL_IMAGE_VERSION: ${MYSQL_IMAGE_VERSION}
    <<: [*restart-always]
    cap_add:
      - SYS_NICE
    security_opt:
      - no-new-privileges:true
    logging:
      options:
        max-size: "50m"
        max-file: "5"
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "--silent"]
      interval: 5s
      timeout: 3s
      retries: 2
      start_period: 0s
    environment:
      MYSQL_DATABASE: ${MYSQL_DATABASE}
      MYSQL_USER: ${MYSQL_USER}
      MYSQL_PASSWORD: ${MYSQL_PASSWORD}
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
    volumes:
      - mysqly:/var/lib/mysql
      - ./data/mysql/init:/docker-entrypoint-initdb.d
    networks:
      - crowdsec

  wait:
    image: waisbrot/wait
    environment:
      TARGETS: mysql:3306 crowdsec:8080
      TIMEOUT: 60
    networks:
      - crowdsec

volumes:
  crowdsec_datax:
    driver: local
  mysqly:
    driver: local
  proxy_logs:
    external: true

networks:
  crowdsec:
    driver: bridge
  proxy:
    external: true

/etc/crowdsec/local_api_credentials.yaml

url: http://0.0.0.0:8080
login: localhost
password: xxx

/etc/crowdsec/config.yaml.local

db_config:
  log_level: info
  type: ${DATABASE_TYPE}
  user: ${DATABASE_USER}
  password: ${DATABASE_USER_PASSWORD}
  db_name: ${DATABASE_NAME}
  host: ${DATABASE_HOST}
  port: ${DATABASE_PORT}
api:
  client:
    insecure_skip_verify: false
    credentials_path: /etc/crowdsec/local_api_credentials.yaml
  server:
    enable: true
    log_level: info
    listen_uri: 0.0.0.0:8080
    trusted_ips:
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 192.168.0.0/16

Anything else we need to know?

I've explicitly mounted every config file separately in my crowdsec service, and not creating a named volume to /etc/crowdsec (like stated in docs) as this would prevent me (the user) from ever updating config settings.

Crowdsec version

$ cscli version
v1.5.3-rc4-debian

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ uname -a
# paste output here
Linux 5.15.0-78-generic #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Enabled collections and parsers

$ cscli hub list -o raw
# N/A

Acquisition config

# On Linux:
$ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/*
filenames:
  - /var/log/auth.log
  - /var/log/syslog
  - /var/log/kern.log
  - /var/log/ufw.log
  - /var/log/mail.log
labels:
  type: syslogfilenames:
  - /log/traefik/*
labels:
  type: traefik

Config show

$ cscli config show
# N/A

Prometheus metrics

$ cscli metrics
# paste output here

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

[github-actions[bot]]

@ToshY: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[mmetc]

Thanks

so what happens if you just don't mount local_api_credentials.yaml?

The "localhost" machine would be recreated every time the container is started, but it's done on purpose. Does it work for you?

[ToshY]

Hey @mmetc 👋

Okay I think this might be a major misunderstanding on my side.

I'm confused what the usage is of local_api_credentials.yaml, and I tried to use the password from local_api_credentials.yaml for the Firewall Bouncer (at the api_key). That's why I wanted to mount it in order to keep the same password. But what I think I'm actually supposed to do is, add a bouncer instead, and use that api key for my firewall bouncer. 🤦

If that's correct, could you still maybe explain me a couple of things:

1. Where are the bouncer API keys stored? Is it (encrypted) in the database?
2. This part in the documentation shows an example (and text explanation) for mounting of local_api_credentials.yaml. So I'm not sure if it should still be possible or not?

[mmetc]

The local_api_credentials.yaml file is used by cscli and by the log processor (agent). So when you have both lapi and log processor enabled, which is the case by default, crowdsec actually authenticated to itself via a "local" machine. There is a PR to use a unix socket instead but has not been merged yet.

1. yes, it's in the db
2. the convenience in that case is to use cscli from outside the container. It's equally possible to call "docker exec -ti cscli...." and why not, with a shell alias. We should probably amend the documentation.. I'll have a look at the use case for MacOS and see to fix it. thanks