[hub] introduce cscli hub fix command

[LaurenceJJones]

There has been an outstanding issue for a long time if a user "accidentally" install the debian packages and then upgrades to our repository version all symlinks point towards deleted files.

An idea could be to have cscli hub fix command that goes through the current symlinks and tries to rectify the broken symlinks for example after installing the debian package and upgrading to our this is the output of cscli parsers list

root@bookworm:~# cscli parsers list
WARN link target does not exist: /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml -> /var/lib/crowdsec/hub/parsers/s00-raw/crowdsecurity/syslog-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s01-parse/apache2-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/apache2-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/nginx-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/sshd-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml -> /var/lib/crowdsec/hub/parsers/s02-enrich/crowdsecurity/dateparse-enrich.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s02-enrich/http-logs.yaml -> /var/lib/crowdsec/hub/parsers/s02-enrich/crowdsecurity/http-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s02-enrich/whitelists.yaml -> /var/lib/crowdsec/hub/parsers/s02-enrich/crowdsecurity/whitelists.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-26134.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-26134.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-35914.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-35914.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-37042.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-37042.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-40684.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-40684.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-41082.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-41082.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-41697.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-41697.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-42889.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-42889.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-44877.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-44877.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-46169.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-46169.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/apache_log4j2_cve-2021-44228.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/apache_log4j2_cve-2021-44228.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/f5-big-ip-cve-2020-5902.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/f5-big-ip-cve-2020-5902.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/fortinet-cve-2018-13379.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/fortinet-cve-2018-13379.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/grafana-cve-2021-43798.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/grafana-cve-2021-43798.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-backdoors-attempts.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-backdoors-attempts.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-bad-user-agent.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-bad-user-agent.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-crawl-non_statics.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-crawl-non_statics.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-cve-2021-41773.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-cve-2021-41773.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-cve-2021-42013.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-cve-2021-42013.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-generic-bf.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-generic-bf.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-open-proxy.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-open-proxy.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-path-traversal-probing.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-path-traversal-probing.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-probing.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-probing.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-sensitive-files.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-sensitive-files.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-sqli-probing.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-sqli-probing.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-w00tw00t.yaml -> /var/lib/crowdsec/hub/scenarios/ltsich/http-w00tw00t.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-xss-probing.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-xss-probing.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/jira_cve-2021-26086.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/jira_cve-2021-26086.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/nginx-req-limit-exceeded.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/nginx-req-limit-exceeded.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/pulse-secure-sslvpn-cve-2019-11510.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/pulse-secure-sslvpn-cve-2019-11510.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/spring4shell_cve-2022-22965.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/spring4shell_cve-2022-22965.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/ssh-bf.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/ssh-bf.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/ssh-slow-bf.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/ssh-slow-bf.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/thinkphp-cve-2018-20062.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/thinkphp-cve-2018-20062.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/vmware-cve-2022-22954.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/vmware-cve-2022-22954.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/vmware-vcenter-vmsa-2021-0027.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/vmware-vcenter-vmsa-2021-0027.yaml
WARN link target does not exist: /etc/crowdsec/collections/apache2.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/apache2.yaml
WARN link target does not exist: /etc/crowdsec/collections/base-http-scenarios.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/base-http-scenarios.yaml
WARN link target does not exist: /etc/crowdsec/collections/http-cve.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/http-cve.yaml
WARN link target does not exist: /etc/crowdsec/collections/linux.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/linux.yaml
WARN link target does not exist: /etc/crowdsec/collections/nginx.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/nginx.yaml
WARN link target does not exist: /etc/crowdsec/collections/sshd.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/sshd.yaml

PARSERS
──────────────────────────────────────
Name  📦 Status  Version  Local Path
──────────────────────────────────────
──────────────────────────────────────

I did used to have a script https://gist.github.com/LaurenceJJones/6960107296145e8e365009973b9d7f6d that would fix this, however, with recent changes to the hub no items are displayed and it cannot be fixed like this anymore.

Edit: This will improve the user experience then having to completely remove the package (potentially remove there own custom configuration) and having to restart all over again if they happen to notice this late in the process.

[github-actions[bot]]

@LaurenceJJones: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions[bot]]

@LaurenceJJones: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[bakrhaso]

In this situation right now, having a command to fix it would be nice!

[OfficialMuffin]

Reinstalled crowdsec after a while, getting the same list of warnings.

UPDATE:

Uninstalling crowdsec and bouncers via APT, installing locate, running updatedb, then running locate to find all the residual crowdsec files. Then going through and manually deleting the residual files. After that, reinstall crowdsec and the bouncer. This fixes this issue for me and is most likely the current workaround.

[mmetc]

@LaurenceJJones I tried to go down the route of "cscli hub fix", noting that

• the command runs when the hub content from the previous package (1.4.6) is already removed. we have the links but no source or index, no consistent state to copy or reinstall
• running it in pre-remove has its own issues, like there is an old hub but not the new one, or index
• adding that in cscli would make the code unnecessary more complex when it can be done outside of the process, in bash

So I wrote a script that can be run right after installing 1.6.5 (not tested with 1.6.4), here

https://raw.githubusercontent.com/crowdsecurity/crowdsec/refs/heads/migrate-hub/debian/migrate-hub.sh

#!/usr/bin/env sh

set -eu

# Download everything on the new hub but don't install anything yet

echo "Downloading Hub content..."

for itemtype in $(cscli hub types -o raw); do
    ALL_ITEMS=$(cscli "$itemtype" list -a -o raw | tail +2 | cut -d, -f1)
    if [ -n "${ALL_ITEMS}" ]; then
        # shellcheck disable=SC2086
        cscli "$itemtype" install \
            $ALL_ITEMS \
            --download-only -y
    fi
done

# Fix links

BASEDIR=/etc/crowdsec/
OLD_PATH=/var/lib/crowdsec/hub/
NEW_PATH=/etc/crowdsec/hub/

find "$BASEDIR" -type l 2>/dev/null | while IFS= read -r link
do
    target="$(readlink "$link")" || continue

    case "$target" in
        "$OLD_PATH"*)
            suffix="${target#"$OLD_PATH"}"
            new_target="${NEW_PATH}${suffix}"

            if [ -e "$target" ]; then
                continue
            fi

            if [ ! -e "$new_target" ]; then
                continue
            fi

            echo "Update symlink: $link"
            ln -sf "$new_target" "$link"
            ;;
        *)
            ;;
    esac
done

# upgrade tainted collections

cscli hub upgrade --force

with the following caveats

• it takes some time and downloads all the hub and data files
• it's very verbose and doesn't hide the warnings until it's finished
• tainted stuff will be replaced.

Otherwise it should do the job. I'm not sure if it's a good idea to install and run it automatically in its current state.