k8s - hash mismatch on persistent volume

[usma0118]

What happened?

crowdsec agent startup failed with following error:

Defaulted container "crowdsec-agent" out of: crowdsec-agent, wait-for-lapi (init)
/etc/crowdsec_data was found in a volume
Running hub update
Skipping hub update, index file is recent
Skipping hub upgrade, data directory is not in a volume
Running: cscli  parsers install "crowdsecurity/docker-logs"
installed crowdsecurity/docker-logs
level=info msg="Enabled crowdsecurity/docker-logs"
level=info msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
Running: cscli  parsers install "crowdsecurity/cri-logs"
installed crowdsecurity/cri-logs
level=info msg="Enabled crowdsecurity/cri-logs"
level=info msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
Running: cscli  collections install "crowdsecurity/nginx"
level=fatal msg="error while installing 'crowdsecurity/nginx': while downloading crowdsecurity/nginx-logs: while downloading crowdsecurity/nginx-logs to https://hub-cdn.crowdsec.net/master/parsers/s01-parse/crowdsecurity/nginx-logs.yaml: hash mismatch: expected 1948e74edab6e6fa23f70675e2883b726d4e0394314dafaad2b9819762b92b34, got 538990ce5b01974ddd29c948de56322b92de56f6d9e70fc7f45415ce8af3858d"
Failed to install collections/crowdsecurity/nginx, running hub update before retrying
Skipping hub update, index file is recent
level=fatal msg="error while installing 'crowdsecurity/nginx': while downloading crowdsecurity/nginx-logs: while downloading crowdsecurity/nginx-logs to https://hub-cdn.crowdsec.net/master/parsers/s01-parse/crowdsecurity/nginx-logs.yaml: hash mismatch: expected 1948e74edab6e6fa23f70675e2883b726d4e0394314dafaad2b9819762b92b34, got 538990ce5b01974ddd29c948de56322b92de56f6d9e70fc7f45415ce8af3858d"

What did you expect to happen?

for crowdsect agent to startup and load collections.

How can we reproduce it (as minimally and precisely as possible)?

upgrade running helm version: 0.10.0 to version: 0.11.0

Anything else we need to know?

No response

Crowdsec version

$ cscli version
# paste output here

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Enabled collections and parsers

$ cscli hub list -o raw
# paste output here

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* filenames: - /var/log/nginx/*.log - ./tests/nginx/nginx.log #this is not a syslog log, indicate which kind of logs it is labels: type: nginx --- filenames: - /var/log/auth.log - /var/log/syslog labels: type: syslog --- filename: /var/log/apache2/*.log labels: type: apache2

On Windows:

C:> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml

paste output here

Config show

$ cscli config show
E0827 09:26:12.661835   65905 websocket.go:296] Unknown stream id 1, discarding message
                                                                                       Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log
   - Log level              : info
   - Log Media              : stdout
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              :
API Client:
  - URL                     : http://localhost:8080/
  - Login                   : localhost
  - Credentials File        : /etc/crowdsec/local_api_credentials.yaml
Local API Server:
  - Listen URL              : 0.0.0.0:8080
  - Listen Socket           :
  - Profile File            : /etc/crowdsec/profiles.yaml

  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : sqlite
      - Path                : /var/lib/crowdsec/data/crowdsec.db
      - Flush age           : 7d
      - Flush size          : 5000

Prometheus metrics

$ cscli metrics
╭────────────────────────────────────────────┬────────┬────────┬───────╮
│                   Reason                   │ Origin │ Action │ Count │
├────────────────────────────────────────────┼────────┼────────┼───────┤
│ crowdsecurity/http-sensitive-files         │ CAPI   │ ban    │ 216   │
│ crowdsecurity/apache_log4j2_cve-2021-44228 │ CAPI   │ ban    │ 47    │
│ crowdsecurity/http-bad-user-agent          │ CAPI   │ ban    │ 8248  │
│ crowdsecurity/nginx-req-limit-exceeded     │ CAPI   │ ban    │ 397   │
│ crowdsecurity/thinkphp-cve-2018-20062      │ CAPI   │ ban    │ 84    │
│ crowdsecurity/http-generic-bf              │ CAPI   │ ban    │ 25    │
│ crowdsecurity/http-path-traversal-probing  │ CAPI   │ ban    │ 146   │
│ crowdsecurity/netgear_rce                  │ CAPI   │ ban    │ 15    │
│ crowdsecurity/CVE-2019-18935               │ CAPI   │ ban    │ 19    │
│ crowdsecurity/http-crawl-non_statics       │ CAPI   │ ban    │ 314   │
│ crowdsecurity/ssh-slow-bf                  │ CAPI   │ ban    │ 7957  │
│ crowdsecurity/http-backdoors-attempts      │ CAPI   │ ban    │ 148   │
│ crowdsecurity/http-cve-probing             │ CAPI   │ ban    │ 8     │
│ crowdsecurity/CVE-2023-22515               │ CAPI   │ ban    │ 3     │
│ crowdsecurity/pgsql-bf                     │ CAPI   │ ban    │ 21    │
│ crowdsecurity/CVE-2022-35914               │ CAPI   │ ban    │ 2     │
│ crowdsecurity/CVE-2022-37042               │ CAPI   │ ban    │ 2     │
│ crowdsecurity/CVE-2022-26134               │ CAPI   │ ban    │ 9     │
│ crowdsecurity/http-cve-2021-41773          │ CAPI   │ ban    │ 187   │
│ ltsich/http-w00tw00t                       │ CAPI   │ ban    │ 3     │
│ crowdsecurity/fortinet-cve-2018-13379      │ CAPI   │ ban    │ 10    │
│ crowdsecurity/http-admin-interface-probing │ CAPI   │ ban    │ 228   │
│ crowdsecurity/http-cve-2021-42013          │ CAPI   │ ban    │ 3     │
│ crowdsecurity/http-open-proxy              │ CAPI   │ ban    │ 1435  │
│ crowdsecurity/http-probing                 │ CAPI   │ ban    │ 2991  │
│ crowdsecurity/http-wordpress-scan          │ CAPI   │ ban    │ 236   │
│ crowdsecurity/jira_cve-2021-26086          │ CAPI   │ ban    │ 15    │
│ crowdsecurity/ssh-bf                       │ CAPI   │ ban    │ 6351  │
│ crowdsecurity/CVE-2017-9841                │ CAPI   │ ban    │ 161   │
│ crowdsecurity/CVE-2023-49103               │ CAPI   │ ban    │ 85    │
│ crowdsec_paris_2024_intelligence           │ lists  │ ban    │ 6455  │
╰────────────────────────────────────────────┴────────┴────────┴───────╯

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

[github-actions[bot]]

@usma0118: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[usma0118]

already tried #2946 without any success.

[LaurenceJJones]

already tried #2946 without any success.

Can you provide the docker run command that was executed?

[usma0118]

already tried #2946 without any success.

Can you provide the docker run command that was executed?

I am using k8s with helm, not docker. if you want to see chart values, those can be seen here: https://pastebin.com/mAZgwKV5

[blotus]

Hello,

We are working on a long-term fix for this type of issue that should be part of 1.6.3, but in the meantime, you can try the following:

• Create a file pod.yaml with the following content (make sure the crowdsec version specified is the same as the one you use):

apiVersion: v1
kind: Pod
metadata:
 name: temp-cscli-update
spec:
 containers:
 - name: temp-cscli-update
   image: crowdsecurity/crowdsec:v1.6.2
   command: ["sh", "-c", "ln -s /etc/crowdsec_data /etc/crowdsec && cscli hub update"]
   volumeMounts:
   - mountPath: /etc/crowdsec_data
     name: crowdsec-agent-config
 volumes:
 - name: crowdsec-agent-config
   persistentVolumeClaim:
     claimName: crowdsec-agent-config-pvc
 restartPolicy: Never

• Run this pod: kubectl apply -f pod.yaml
• It should take just a few seconds, you can check the hub was updated properly with kubectl logs temp-cscli-update
• If it succeeded, you can delete the pod: kubectl delete -f pod.yaml

[LaurenceJJones]

@usma0118 Did the provided workaround managed to fix the issue? if not please reopen the issue and provided relevant details as to why it did not work or resolve the issue