lemmy-docs icon indicating copy to clipboard operation
lemmy-docs copied to clipboard

Document that high db pool size requires increased shared memory for postgres

Open Nutomic opened this issue 2 years ago • 4 comments

Setting a high db pool size causes various sql query errors, most notably logins failing entirely. To fix it, shared memory for postgres needs to be increased with the following line in docker-compose.yml. This should be documented, but not sure where it fits.

postgres:
    image: postgres:15-alpine
    shm_size: 1g
    ...

Original Lemmy issue

Logins on lemmy.ml are consistently failing with known working user/password. Server logs show the following error:

lemmy_1     | 2023-07-26T12:52:38.922771Z  WARN lemmy_server::root_span_builder: incorrect_login: could not resize shared memory segment "/PostgreSQL.1513506408" to 220224 bytes: No space left on device
lemmy_1     |    0: lemmy_api::local_user::login::perform
lemmy_1     |            with self=Login { username_or_email: Sensitive, password: Sensitive, totp_2fa_token: None }
lemmy_1     |              at crates/api/src/local_user/login.rs:16
lemmy_1     |    1: lemmy_server::root_span_builder::HTTP request
lemmy_1     |            with http.method=POST http.scheme="http" http.host=lemmy.ml http.target=/api/v3/user/login otel.kind="server" request_id=1853cb5f-c9c7-4794-bf02-4bbef2ed5c41
lemmy_1     |              at src/root_span_builder.rs:16
lemmy_1     | LemmyError { message: Some("incorrect_login"), inner: could not resize shared memory segment "/PostgreSQL.1513506408" to 220224 bytes: No space left on device, context: SpanTrace [{ target: "lemmy_api::local_user::login", name: "perform", fields: "self=Login { username_or_email: Sensitive, password: Sensitive, totp_2fa_token: None }", file: "crates/api/src/local_user/login.rs", line: 16 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"http\" http.host=lemmy.ml http.target=/api/v3/user/login otel.kind=\"server\" request_id=1853cb5f-c9c7-4794-bf02-4bbef2ed5c41", file: "src/root_span_builder.rs", line: 16 }] }

A lemmy restart fixes the problem temporarily. There is plenty of free disk space on the server.

The problem might be related to the fact that I set the db pool size to 95 because of LemmyNet/lemmy#3661. I lowered it to the previous value of 32 now, lets see if the problem happens again.

Version: 0.18.2

Nutomic avatar Jul 26 '23 12:07 Nutomic

could not resize shared memory segment "/PostgreSQL.1513506408" is a problem with shared memory (/dev/shm) not disk space. Is postgres constrained in the amount of shared memory it can use, or is there only a small amount on the server?

daviessm avatar Jul 26 '23 13:07 daviessm

df -h shows this (same if executed inside the postgres docker container):

tmpfs            16G     0   16G   0% /dev/shm

Also it looks like lowering db pool size really did fix the issue.

Nutomic avatar Jul 26 '23 13:07 Nutomic

It's a bit old but have a look at the docker settings: https://stackoverflow.com/questions/56751565/pq-could-not-resize-shared-memory-segment-no-space-left-on-device

daviessm avatar Jul 26 '23 14:07 daviessm

Thanks, that works! I added the following to docker-compose.yml as suggested in the stackoverflow link.

postgres:
    image: postgres:15-alpine
    shm_size: 1g
    ...

Gonna move this to the docs repo.

Nutomic avatar Jul 27 '23 12:07 Nutomic