Unhealthy LDAP Outpost / no connection

[boesr]

Describe the bug I created an LDAP outpost like described at https://goauthentik.io/docs/providers/ldap/generic_setup with the default flow. The outpost container only gets created when I restart the Authentik-Stack (docker compose down && docker compose up -d). It is directly shown as unhealthy and does not allow any connection. Below you find the log of that container.

To Reproduce Steps to reproduce the behavior:

1. Create LDAP Outpost / Provider according to documentation

Expected behavior Healthy outpost container and able to connect e.g. via ldapsearch

Screenshots /

Logs

Log of the ldap outpost container.

{"event":"Loaded config","level":"debug","path":"inbuilt-default","timestamp":"2024-01-25T13:16:45Z"}
{"event":"Loaded config from environment","level":"debug","timestamp":"2024-01-25T13:16:45Z"}
{"event":"not enabling debug server, set `AUTHENTIK_DEBUG` to `true` to enable it.","level":"info","logger":"authentik.go_debugger","timestamp":"2024-01-25T13:16:45Z"}

Version and Deployment (please complete the following information):

• authentik version: 2023.10.6]
• Deployment: docker-compose

[boesr]

The container not being created seems to be connected to LDAP sync. As soon as I deactivate the LDAP source sync (close to 100k users, 10 workers active), the container gets created. It is still unhealthy though.

Another hint: The issue changed, when I start the compose file with only one worker. Then I receive:

{"error":"Get \"https://myserver.com/api/v3/core/users/?page=53&page_size=100\": dial tcp 123.45.67.899:443: connect: connection refused","event":"failed to update users","level":"warning","timestamp":"2024-01-25T14:43:26Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=53&page_size=100\": dial tcp 123.45.67.899:443: connect: connection refused","event":"Failed to fetch user page","level":"warning","page":53,"timestamp":"2024-01-25T14:43:26Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=53&page_size=100\": dial tcp 123.45.67.899:443: connect: connection refused","event":"failed to update users","level":"warning","timestamp":"2024-01-25T14:43:26Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=53&page_size=100\": dial tcp 123.45.67.899:443: connect: connection refused","event":"Failed to fetch user page","level":"warning","page":53,"timestamp":"2024-01-25T14:43:26Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=53&page_size=100\": dial tcp 123.45.67.899:443: connect: connection refused","event":"failed to update users","level":"warning","timestamp":"2024-01-25T14:43:26Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=53&page_size=100\": dial tcp 123.45.67.899:443: connect: connection refused","event":"Failed to fetch user page","level":"warning","page":53,"timestamp":"2024-01-25T14:43:26Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=53&page_size=100\": dial tcp 123.45.67.899:443: connect: connection refused","event":"failed to update users","level":"warning","timestamp":"2024-01-25T14:43:26Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=53&page_size=100\": dial tcp 123.45.67.899:443: connect: connection refused","event":"Failed to fetch user page","level":"warning","page":53,"timestamp":"2024-01-25T14:43:26Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=53&page_size=100\": dial tcp 123.45.67.899:443: connect: connection refused","event":"failed to update users","level":"warning","timestamp":"2024-01-25T14:43:26Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=53&page_size=100\": dial tcp 123.45.67.899:443: connect: connection refused","event":"Failed to fetch user page","level":"warning","page":53,"timestamp":"2024-01-25T14:43:26Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=53&page_size=100\": dial tcp 123.45.67.899:443: connect: connection refused","event":"failed to update users","level":"warning","timestamp":"2024-01-25T14:43:26Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=53&page_size=100\": dial tcp 123.45.67.899:443: connect: connection refused","event":"Failed to fetch user page","level":"warning","page":53,"timestamp":"2024-01-25T14:43:26Z"}
{"error":"502 Bad Gateway","event":"Failed to fetch outpost configuration, retrying in 3 seconds","level":"error","logger":"authentik.outpost.ak-api-controller","timestamp":"2024-01-25T14:43:51Z"}

since above error indicates some cert problem I set authentik_host_insecure: false, which results in the following:

{"error":"Get \"https://myserver.com/api/v3/core/users/?page=90&page_size=100\": read tcp 10.10.0.2:44058->123.45.67.899:443: read: connection reset by peer","event":"failed to update users","level":"warning","timestamp":"2024-01-25T14:50:49Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=90&page_size=100\": read tcp 10.10.0.2:44058->123.45.67.899:443: read: connection reset by peer","event":"Failed to fetch user page","level":"warning","page":90,"timestamp":"2024-01-25T14:50:49Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=90&page_size=100\": read tcp 10.10.0.2:44070->123.45.67.899:443: read: connection reset by peer","event":"failed to update users","level":"warning","timestamp":"2024-01-25T14:50:49Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=90&page_size=100\": read tcp 10.10.0.2:44070->123.45.67.899:443: read: connection reset by peer","event":"Failed to fetch user page","level":"warning","page":90,"timestamp":"2024-01-25T14:50:49Z"}
{"error":"Get \"https://myserver.com/api/v3/core/users/?page=90&page_size=100\": read tcp 10.10.0.2:44084->123.45.67.899:443: read: connection reset by peer","event":"failed to update users","level":"warning","timestamp":"2024-01-25T14:50:49Z"}

Further testing with only one worker enabled: The outpost container does get healthy around 2 hours after creation. Can I somehow improve that? The log of the ldap outpost container:

{"event":"Loaded config","level":"debug","path":"inbuilt-default","timestamp":"2024-01-26T07:35:26Z"}
{"event":"Loaded config from environment","level":"debug","timestamp":"2024-01-26T07:35:26Z"}
{"event":"not enabling debug server, set `AUTHENTIK_DEBUG` to `true` to enable it.","level":"info","logger":"authentik.go_debugger","timestamp":"2024-01-26T07:35:26Z"}
{"event":"Fetching certificate and private key","level":"info","logger":"authentik.outpost.cryptostore","timestamp":"2024-01-26T07:35:27Z","uuid":"c6daa978-53c8-41aa-9f9b-8327dd7b2dad"}
{"event":"Fingerprint hasn't changed, not fetching cert","level":"info","logger":"authentik.outpost.cryptostore","timestamp":"2024-01-26T07:35:27Z","uuid":"c6daa978-53c8-41aa-9f9b-8327dd7b2dad"}
{"event":"initialised direct binder","level":"info","logger":"authentik.outpost.ldap.binder.direct","timestamp":"2024-01-26T10:01:50Z"}
{"event":"Update providers","level":"info","logger":"authentik.outpost.ldap","timestamp":"2024-01-26T10:01:50Z"}
{"event":"Starting Metrics server","level":"info","listen":"0.0.0.0:9300","logger":"authentik.outpost.metrics","timestamp":"2024-01-26T10:01:50Z"}
{"error":"websocket: close 1012","event":"ws read error","level":"warning","logger":"authentik.outpost.ak-api-controller","loop":"ws-handler","timestamp":"2024-01-26T10:01:50Z"}
{"event":"Starting LDAP SSL server","level":"info","listen":"0.0.0.0:6636","logger":"authentik.outpost.ldap","timestamp":"2024-01-26T10:01:50Z"}
{"event":"Starting LDAP server","level":"info","listen":"0.0.0.0:3389","logger":"authentik.outpost.ldap","timestamp":"2024-01-26T10:01:50Z"}
{"event":"Starting authentik outpost","hash":"tagged","level":"info","logger":"authentik.outpost","timestamp":"2024-01-26T10:01:50Z","version":"2023.10.6"}
{"event":"Fingerprint hasn't changed, not fetching cert","level":"info","logger":"authentik.outpost.cryptostore","timestamp":"2024-01-26T10:01:50Z","uuid":"c6daa978-53c8-41aa-9f9b-8327dd7b2dad"}
{"event":"initialised memory searcher","level":"debug","logger":"authentik.outpost.ldap.searcher.memory","timestamp":"2024-01-26T10:01:50Z"}

Afterwards it does work. But 2 hours for a restart is quite long.

[dginhoux]

Hi,

I've the same behavior. I'm trying authentik and have following the same guide for ldap in the docs. I use and add a custom ldap-client+nc service to test it.

It appear there no listen on bot 6636 or 3389 opened on the server/or worker service. Logs in trace mode don't return any 6636 or 3389 in logs, like for 9000 or 9443. And no error seen in logs when grep for ldap pattern.

I specify ldap listen related vars described in https://goauthentik.io/docs/installation/configuration#listen-settings

EDIT : ldapsearch return error and nc "Connection refused"

[boesr]

Hi,

I've the same behavior. I'm trying authentik and have following the same guide for ldap in the docs. I use and add a custom ldap-client+nc service to test it.

It appear there no listen on bot 6636 or 3389 opened on the server/or worker service. Logs in trace mode don't return any 6636 or 3389 in logs, like for 9000 or 9443. And no error seen in logs when grep for ldap pattern.

I specify ldap listen related vars described in https://goauthentik.io/docs/installation/configuration#listen-settings

EDIT : ldapsearch return error and nc "Connection refused"

@dginhoux The ldap port won't open on the server/worker. It will instead create a new container with the ldap server. The ports in the configuration will be applied to the container. See https://goauthentik.io/docs/providers/ldap/generic_setup

[boesr]

After starting a separate ldap outpost container in an interactive session it seems like the ldap container first tries to fetch every existing user. The first few ones went relatively fast. But at the moment (around page 247 with each page having 100 users) ist takes 5 to 10 seconds per page. Since we have close to 100k users that's why it takes hours for the ldap provider to load, I assume. Is there a way to speed up the process?

EDIT:

Everyone that has the same problem. Set Search Mode to direct querying. Then users aren't cached, but it does work.

Would be great if we could mix those -> start ldap with direct querying while the cache is built or something like that

[dginhoux]

Hi, thank for yours answers.

So if i understand, server will take the decision to create new container alone for ldap ? How to handle this in a docker env "user-remaps" and "socketless"... ?

[boesr]

Hi, thank for yours answers.

So if i understand, server will take the decision to create new container alone for ldap ? How to handle this in a docker env "user-remaps" and "socketless"... ?

Not sure about that. You can create the container manually and it should be able to communicate with the Authentik server (since it uses the Rest API)

docker run -e AUTHENTIK_TOKEN=your-token -e AUTHENTIK_HOST=https://your-host.com -d -p 636:6636 --name=ak-outpost-ldap ghcr.io/goauthentik/ldap:2023.10.7

But not sure if you can further manage the container through the Web UI since the server probably cannot see the container (since you use socketless environment).

[boesr]

Unfortunately, setting search mode to direct querying does not solve it completely. The ldap container continuously queries the user pages (like in cached mode) which leads to a 100% cpu usage of the authentik server. I will try to increase the cores it can use (at the moment 8, but went up to 20 when tested before).

Is there a way to stop the ldap server continuously requesting the user data and only request it a specific user, if a request is sent to the ldap server?

logs of server:

{"auth_via": "api_token", "event": "/api/v3/core/users/?page=785&page_size=100", "host": "auth.mydomain.com", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 232923, "remote": "10.10.26.1", "request_id": "0013076b2e1243969ce61dd633847a14", "runtime": 53599, "scheme": "https", "status": 200, "timestamp": "2024-02-01T08:01:55.918403", "user": "ak-outpost-2191069e1a3549808c5456eec3758afd", "user_agent": "goauthentik.io/outpost/2023.10.7"}

Provider search and bind mode are both set to direct querying.

[dginhoux]

Hi, thank for yours answers. So if i understand, server will take the decision to create new container alone for ldap ? How to handle this in a docker env "user-remaps" and "socketless"... ?

Not sure about that. You can create the container manually and it should be able to communicate with the Authentik server (since it uses the Rest API)

docker run -e AUTHENTIK_TOKEN=your-token -e AUTHENTIK_HOST=https://your-host.com -d -p 636:6636 --name=ak-outpost-ldap ghcr.io/goauthentik/ldap:2023.10.7

But not sure if you can further manage the container through the Web UI since the server probably cannot see the container (since you use socketless environment).

Hi,

Thank you for your answer ! You help me a lot.

I also discover and understand the button on the right of outpost "view deploy info" and this doc: https://version-2023-10.goauthentik.io/docs/outposts#deploy

Architecture and container used are now easier to undertsand for me. LDAP work fine now.

Now my next step will to integrate authentike in traefik like authelia.

Have a good day

[authentik-automation[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.