crowdsec icon indicating copy to clipboard operation
crowdsec copied to clipboard

[LAPI] not compatible with libpam-tmpdir package

Open GNU-Plus-Windows-User opened this issue 6 months ago • 2 comments

What happened?

libpam-tmpdir is designed to improve the security of how temp files are stored by creating a dedicated temp directory per user, per session. When using this package on the CrowdSec LAPI, email notifications don't work. Running crowdsec -t gives this output:

FATAL api server init: unable to run plugin broker: while loading plugin: Unrecognized remote plugin message:
This usually means
  the plugin was not compiled for this architecture,
  the plugin is missing dynamic-link libraries necessary to run,
  the plugin is not executable by this process due to file permissions, or
  the plugin failed to negotiate the initial go-plugin protocol handshake

Additional notes about plugin:
  Path: /usr/lib/crowdsec/plugins/notification-email
  Mode: -rwxr-xr-x
  Owner: 0 [root] (current: 0 [root])
  Group: 0 [root] (current: 0 [root])
  ELF architecture: EM_X86_64 (current architecture: amd64)

File permissions are configured correctly and removing the package fixes the issue.

What did you expect to happen?

crowdsec -t should pass with no errors and I should be able to send email notifications.

How can we reproduce it (as minimally and precisely as possible)?

  1. Configure email notifications
  2. install the libpam-tmpdir package
  3. Restart the server
  4. Run crowdsec -t or send a test email notification

Anything else we need to know?

No response

Crowdsec version

version: v1.6.8-debian-pragmatic-amd64-f209766e
Codename: alphaga
BuildDate: 2025-03-25_14:51:10
GoVersion: 1.24.1
Platform: linux
libre2: C++
User-Agent: crowdsec/v1.6.8-debian-pragmatic-amd64-f209766e-linux
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0
Built-in optional components: cscli_setup, datasource_appsec, datasource_cloudwatch, datasource_docker, datasource_file, datasource_http, datasource_journalctl, datasource_k8s-audit, datasource_kafka, datasource_kinesis, datasource_loki, datasource_s3, datasource_syslog, datasource_victorialogs, datasource_wineventlog```

</details>


### OS version

<details>

```console
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Linux lapi 6.8.0-60-generic #63-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 15 19:04:15 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Enabled collections and parsers

N/A

Acquisition config

N/A

Config show

Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log
   - Log level              : info
   - Log Media              : file
Crowdsec:
  - Acquisition File        :
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              :
API Client:
  - URL                     : https://example.com/
  - Login                   : crowdsec-lapi
  - Credentials File        : /etc/crowdsec/local_api_credentials.yaml
Local API Server:
  - Listen URL              : 0.0.0.0:443
  - Listen Socket           :
  - Profile File            : /etc/crowdsec/profiles.yaml
  - Cert File : /etc/letsencrypt/live/example.com/fullchain.pem
  - Key File  : /etc/letsencrypt/live/example.com/privkey.pem

  - Trusted IPs:
      - 127.0.0.1
  - Database:
      - Type                : sqlite
      - Path                : /var/lib/crowdsec/data/crowdsec.db
      - Flush age           : 120d
      - Flush size          : 20000```

</details>


### Prometheus metrics

<details>

N/A

</details>


### Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

<details>

N/A

</details>

GNU-Plus-Windows-User avatar Jun 16 '25 17:06 GNU-Plus-Windows-User

@GNU-Plus-Windows-User: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

github-actions[bot] avatar Jun 16 '25 17:06 github-actions[bot]

Yes this is most likely cause the gRPC package we use for notifications uses the temp directory to place Unix sockets for IPC communication between crowdsec and the plugins. (And if it crashes we don't have to worry about cleaning up dead sockets cause when you restart they will be wiped)

I don't know if we can override this behaviour and aim for a general location like /run but we don't know if they will have this directory but they most likely will have a temp dir.

LaurenceJJones avatar Jun 16 '25 18:06 LaurenceJJones

The plugin package will use $PLUGIN_UNIX_SOCKET_DIR instead of /tmp, if that variable exists.

Unfortunately, the version we use is too old atm, so I propose to update and test: https://github.com/crowdsecurity/crowdsec/issues/3688

mmetc avatar Jun 20 '25 12:06 mmetc

@mmetc @LaurenceJJones I've upgraded to 1.6.10 and I still have the exact same error message with libpam-tmpdir installed.

GNU-Plus-Windows-User avatar Jul 17 '25 00:07 GNU-Plus-Windows-User

@mmetc @LaurenceJJones I've upgraded to 1.6.10 and I still have the exact same error message with libpam-tmpdir installed.

Yes we was not clear, we updated the packages this means you can specify PLUGIN_UNIX_SOCKET_DIR environment variable which allows you to move the sockets outside the tmp directory to another folder like run.

LaurenceJJones avatar Jul 17 '25 10:07 LaurenceJJones

@LaurenceJJones I'm not using docker so how would I change the socket directory for the email plugin? I don't see it anywhere in the docs

GNU-Plus-Windows-User avatar Jul 20 '25 11:07 GNU-Plus-Windows-User

@LaurenceJJones Where do I specify the PLUGIN_UNIX_SOCKET_DIR is this in some config file I need to edit? It's not clear what I need to do to fix the compatibility issue with libpam-tmpdir

GNU-Plus-Windows-User avatar Aug 25 '25 08:08 GNU-Plus-Windows-User

@LaurenceJJones Where do I specify the PLUGIN_UNIX_SOCKET_DIR is this in some config file I need to edit? It's not clear what I need to do to fix the compatibility issue with libpam-tmpdir

Where are you running CrowdSec?

If via traditional systemd files you would create an override file via systemctl edit crowdsec.service.

  1. Open an override for the service

sudo systemctl edit crowdsec.service

This opens an editor (usually nano or vim) for a drop-in file like /etc/systemd/system/crowdsec.service.d/override.conf.


  1. Add the environment variable

In the file, add an [Service] section and your variable:

[Service] Environment="PLUGIN_UNIX_SOCKET_DIR=/path/to/directory"

(Replace /path/to/directory with the actual directory you want CrowdSec plugins to use.)


  1. Reload systemd to pick up the change

sudo systemctl daemon-reexec

(or daemon-reload — either is fine here)


  1. Restart CrowdSec

sudo systemctl restart crowdsec.service


  1. Verify it took effect

You can confirm the environment variable is set with:

systemctl show crowdsec.service --property=Environment

LaurenceJJones avatar Aug 25 '25 08:08 LaurenceJJones

@LaurenceJJones

I created a drop-in file for systemd but I'm still having the same issue:

[Service]
Environment="PLUGIN_UNIX_SOCKET_DIR=/crowdsec-plugins/"

crowdsec -t gives the same error message:

FATAL api server init: plugin broker: loading plugin: Unrecognized remote plugin message:
Failed to read any lines from plugin's stdout
This usually means
  the plugin was not compiled for this architecture,
  the plugin is missing dynamic-link libraries necessary to run,
  the plugin is not executable by this process due to file permissions, or
  the plugin failed to negotiate the initial go-plugin protocol handshake

Additional notes about plugin:
  Path: /usr/lib/crowdsec/plugins/notification-email
  Mode: -rwxr-xr-x
  Owner: 0 [root] (current: 0 [root])
  Group: 0 [root] (current: 0 [root])
  ELF architecture: EM_X86_64 (current architecture: amd64)

I can see my custom directory is being used and the unix socket was created, the directory is set to 777 so permissions shouldn't be an issue

/crowdsec-plugins$ ls -l
total 0
srwxr-xr-x 1 nobody nogroup 0 Aug 27 20:17 plugin3293749128

This is the same as if I was using /tmp, I can see the unix socket is being created there too with the same permissions:

/tmp$ ls -l
total 24
srwxr-xr-x 1 nobody nogroup    0 Aug 27 20:26 plugin4132665906

GNU-Plus-Windows-User avatar Aug 27 '25 10:08 GNU-Plus-Windows-User