langfuse-docs icon indicating copy to clipboard operation
langfuse-docs copied to clipboard
Published 1 month ago • verified publisher icon langfuse

ClickHouse OOM error in docker-compose

Open secretxuan opened this issue 4 months ago • 5 comments

When I use docker engine to deploy langfuse locally, I always encounter clickhouse memory leak problem.

secretxuan avatar Sep 02 '25 02:09 secretxuan

Hi, @secretxuan, can you share a bit more which memory leak you encounter?

maxdeichmann avatar Sep 02 '25 09:09 maxdeichmann

sure，my error docker compose logs are below it might be clickhouse langfuse-web-1 | 2025-08-29T09:06:34.088Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED clickhouse-1 | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log clickhouse-1 | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log clickhouse-1 | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log clickhouse-1 | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log langfuse-web-1 | 2025-08-29T09:06:44.131Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-worker-1 | 2025-08-29T16:00:01.920Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-worker-1 | 2025-08-29T16:00:01.920Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-worker-1 | 2025-08-29T16:00:01.920Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-worker-1 | 2025-08-29T16:00:01.924Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-worker-1 | 2025-08-29T16:00:01.924Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } clickhouse-1 | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log langfuse-worker-1 | 2025-08-29T16:00:01.924Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-29T09:06:45.149Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-web-1 | 2025-08-29T09:06:47.167Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-worker-1 | 2025-08-30T16:00:01.299Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-worker-1 | 2025-08-30T16:00:01.299Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-worker-1 | 2025-08-30T16:00:01.299Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-worker-1 | 2025-08-30T16:00:01.302Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-29T09:06:51.183Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-web-1 | 2025-08-29T09:07:01.215Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-web-1 | 2025-08-29T09:07:02.330Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-web-1 | 2025-08-29T09:07:02.446Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-worker-1 | 2025-08-31T16:00:02.000Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-worker-1 | 2025-08-31T16:00:02.000Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-worker-1 | 2025-08-31T16:00:02.000Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-worker-1 | 2025-08-31T16:00:02.000Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-29T09:07:02.480Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-web-1 | 2025-08-29T09:07:02.492Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-worker-1 | 2025-09-01T08:58:34.866Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 4.72 MiB bytes), current RSS: 2.34 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T08:58:34.866Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 4.72 MiB bytes), current RSS: 2.34 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T09:01:17.907Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.72 MiB bytes), current RSS: 1.59 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T09:01:17.908Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.72 MiB bytes), current RSS: 1.59 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T11:38:38.187Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 2.09 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-web-1 | 2025-08-29T09:07:03.463Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-worker-1 | 2025-09-01T11:38:38.187Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 2.09 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-web-1 | 2025-08-29T09:07:03.497Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-worker-1 | 2025-09-01T11:43:50.317Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 2.53 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T11:43:50.317Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 2.53 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T11:56:38.641Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 4.47 MiB bytes), current RSS: 1.81 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T11:56:38.641Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 4.47 MiB bytes), current RSS: 1.81 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T12:44:31.972Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 0.00 B bytes), current RSS: 2.20 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing AggregatingTransform: while pushing to view default.project_environments_observations_mv: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T12:44:31.972Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 0.00 B bytes), current RSS: 2.20 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing AggregatingTransform: while pushing to view default.project_environments_observations_mv: While executing WaitForAsyncInsert. langfuse-web-1 | 2025-08-29T09:07:03.507Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T13:25:10.058Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 1.83 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T13:25:10.058Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 1.83 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T13:32:07.266Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.59 MiB bytes), current RSS: 1.57 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T13:32:07.266Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.59 MiB bytes), current RSS: 1.57 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T13:39:42.515Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 1.49 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T13:39:42.515Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 1.49 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:00:38.128Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.46 GiB (attempt to allocate chunk of 4.50 MiB bytes), current RSS: 1.28 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:00:38.129Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.46 GiB (attempt to allocate chunk of 4.50 MiB bytes), current RSS: 1.28 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:00:43.130Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 1.12 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:00:43.130Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 1.12 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:00:44.133Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 4.53 MiB bytes), current RSS: 1.18 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:00:44.133Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 4.53 MiB bytes), current RSS: 1.18 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:00:48.133Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.43 GiB (attempt to allocate chunk of 4.53 MiB bytes), current RSS: 1.24 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:00:48.133Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.43 GiB (attempt to allocate chunk of 4.53 MiB bytes), current RSS: 1.24 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:00:49.134Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 1.26 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-web-1 | 2025-08-29T09:07:04.345Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-web-1 | 2025-08-29T09:07:05.480Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-worker-1 | 2025-09-01T14:00:49.134Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 1.26 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:00:50.135Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 1.32 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:00:50.136Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 1.32 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:00:50.136Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:00:50.136Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:00:52.134Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.50 GiB (attempt to allocate chunk of 4.53 MiB bytes), current RSS: 1.34 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:00:52.134Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.50 GiB (attempt to allocate chunk of 4.53 MiB bytes), current RSS: 1.34 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:00:54.136Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 1.23 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:00:54.136Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 1.23 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:00:56.140Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.42 GiB (attempt to allocate chunk of 4.53 MiB bytes), current RSS: 1.20 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:00:56.140Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.42 GiB (attempt to allocate chunk of 4.53 MiB bytes), current RSS: 1.20 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:00:58.138Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.45 GiB (attempt to allocate chunk of 4.55 MiB bytes), current RSS: 1.13 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:00:58.138Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.45 GiB (attempt to allocate chunk of 4.55 MiB bytes), current RSS: 1.13 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:00:59.140Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.43 GiB (attempt to allocate chunk of 4.55 MiB bytes), current RSS: 1.18 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:00:59.140Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.43 GiB (attempt to allocate chunk of 4.55 MiB bytes), current RSS: 1.18 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:01:00.140Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 4.55 MiB bytes), current RSS: 1.19 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:01:00.140Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 4.55 MiB bytes), current RSS: 1.19 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:01:00.140Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:01:00.140Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:16:52.566Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.88 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.20 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:16:52.566Z error ClickhouseWriter.flush traces (total) memory limit exceeded: would use 5.88 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.20 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:16:53.564Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.91 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.23 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:16:53.564Z error ClickhouseWriter.flush traces (total) memory limit exceeded: would use 5.91 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.23 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:16:54.567Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.85 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.15 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:16:54.567Z error ClickhouseWriter.flush traces (total) memory limit exceeded: would use 5.85 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.15 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:16:54.567Z error Max attempts reached for traces record. Dropping record. langfuse-worker-1 | 2025-09-01T14:16:57.570Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.82 GiB (attempt to allocate chunk of 4.49 MiB bytes), current RSS: 1.27 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:16:57.571Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.82 GiB (attempt to allocate chunk of 4.49 MiB bytes), current RSS: 1.27 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:16:58.571Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.88 GiB (attempt to allocate chunk of 4.49 MiB bytes), current RSS: 1.30 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:16:58.571Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.88 GiB (attempt to allocate chunk of 4.49 MiB bytes), current RSS: 1.30 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:16:59.574Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.90 GiB (attempt to allocate chunk of 4.49 MiB bytes), current RSS: 1.27 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:16:59.574Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.90 GiB (attempt to allocate chunk of 4.49 MiB bytes), current RSS: 1.27 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:16:59.574Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:16:59.574Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:01.572Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.80 GiB (attempt to allocate chunk of 4.52 MiB bytes), current RSS: 1.26 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:01.572Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.80 GiB (attempt to allocate chunk of 4.52 MiB bytes), current RSS: 1.26 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:02.572Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.81 GiB (attempt to allocate chunk of 4.52 MiB bytes), current RSS: 1.13 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:02.573Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.81 GiB (attempt to allocate chunk of 4.52 MiB bytes), current RSS: 1.13 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:03.574Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.83 GiB (attempt to allocate chunk of 4.56 MiB bytes), current RSS: 1.14 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:03.574Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.83 GiB (attempt to allocate chunk of 4.56 MiB bytes), current RSS: 1.14 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:03.574Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:03.574Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:04.577Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.91 GiB (attempt to allocate chunk of 4.51 MiB bytes), current RSS: 1.19 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:04.577Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.91 GiB (attempt to allocate chunk of 4.51 MiB bytes), current RSS: 1.19 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:05.576Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.86 GiB (attempt to allocate chunk of 4.51 MiB bytes), current RSS: 1.25 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:05.576Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.86 GiB (attempt to allocate chunk of 4.51 MiB bytes), current RSS: 1.25 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:05.576Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:05.576Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:12.578Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.80 GiB (attempt to allocate chunk of 4.54 MiB bytes), current RSS: 1.20 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:12.579Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.80 GiB (attempt to allocate chunk of 4.54 MiB bytes), current RSS: 1.20 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:13.579Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.96 GiB (attempt to allocate chunk of 4.54 MiB bytes), current RSS: 1.21 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-web-1 | 2025-08-29T09:07:05.512Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-worker-1 | 2025-09-01T14:17:13.579Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.96 GiB (attempt to allocate chunk of 4.54 MiB bytes), current RSS: 1.21 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:14.579Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.80 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.14 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:14.579Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.80 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.14 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:14.579Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:14.579Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:15.581Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.97 GiB (attempt to allocate chunk of 4.53 MiB bytes), current RSS: 1.27 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:15.581Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.97 GiB (attempt to allocate chunk of 4.53 MiB bytes), current RSS: 1.27 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:16.582Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.91 GiB (attempt to allocate chunk of 4.61 MiB bytes), current RSS: 1.32 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:16.583Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.91 GiB (attempt to allocate chunk of 4.61 MiB bytes), current RSS: 1.32 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-web-1 | 2025-08-29T09:07:05.519Z info middleware intercepted error with code UNAUTHORIZED UNAUTHORIZED langfuse-worker-1 | 2025-09-01T14:17:16.583Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:16.583Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:17.582Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.88 GiB (attempt to allocate chunk of 4.55 MiB bytes), current RSS: 1.34 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:17.582Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.88 GiB (attempt to allocate chunk of 4.55 MiB bytes), current RSS: 1.34 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:18.583Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.81 GiB (attempt to allocate chunk of 4.63 MiB bytes), current RSS: 1.28 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:18.583Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.81 GiB (attempt to allocate chunk of 4.63 MiB bytes), current RSS: 1.28 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:18.583Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:18.583Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:19.582Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.80 GiB (attempt to allocate chunk of 4.54 MiB bytes), current RSS: 1.22 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:19.582Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.80 GiB (attempt to allocate chunk of 4.54 MiB bytes), current RSS: 1.22 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:20.585Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.96 GiB (attempt to allocate chunk of 4.61 MiB bytes), current RSS: 1.23 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:20.585Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.96 GiB (attempt to allocate chunk of 4.61 MiB bytes), current RSS: 1.23 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:20.585Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:20.585Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:21.583Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.92 GiB (attempt to allocate chunk of 4.54 MiB bytes), current RSS: 1.18 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:21.583Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.92 GiB (attempt to allocate chunk of 4.54 MiB bytes), current RSS: 1.18 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:22.583Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.88 GiB (attempt to allocate chunk of 4.54 MiB bytes), current RSS: 1.18 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:22.583Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.88 GiB (attempt to allocate chunk of 4.54 MiB bytes), current RSS: 1.18 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:22.583Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:22.583Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:24.583Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.83 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.22 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:24.583Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.83 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.22 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:25.584Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.88 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.23 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-web-1 | 2025-08-29T16:00:01.919Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-29T16:00:01.919Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-worker-1 | 2025-09-01T14:17:25.584Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.88 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.23 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:26.584Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.81 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.22 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:26.584Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.81 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.22 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-web-1 | 2025-08-29T16:00:01.919Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-worker-1 | 2025-09-01T14:17:26.584Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:17:26.584Z error Max attempts reached for observations record. Dropping record. langfuse-web-1 | 2025-08-29T16:00:01.919Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-29T16:00:02.053Z error Redis error connect ECONNREFUSED 172.20.0.2:6379 langfuse-web-1 | 2025-08-29T16:00:02.055Z error Redis error connect ECONNREFUSED 172.20.0.2:6379 langfuse-worker-1 | 2025-09-01T14:17:40.590Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.80 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.22 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:40.590Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.80 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.22 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-web-1 | 2025-08-29T16:00:02.057Z error Redis error connect ECONNREFUSED 172.20.0.2:6379 langfuse-worker-1 | 2025-09-01T14:17:41.590Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.90 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.24 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:41.590Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.90 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.24 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:42.592Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.88 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.18 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:17:42.592Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.88 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.18 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-web-1 | 2025-08-30T16:00:01.295Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-30T16:00:01.295Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-30T16:00:01.295Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-30T16:00:01.295Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-30T16:00:01.297Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-30T16:00:01.408Z error Redis error connect ECONNREFUSED 172.20.0.4:6379 langfuse-web-1 | 2025-08-30T16:00:01.410Z error Redis error connect ECONNREFUSED 172.20.0.4:6379 langfuse-web-1 | 2025-08-30T16:00:01.412Z error Redis error connect ECONNREFUSED 172.20.0.4:6379 langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:17:42.592Z error Max attempts reached for observations record. Dropping record. langfuse-web-1 | 2025-08-31T16:00:02.000Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-31T16:00:02.001Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-31T16:00:02.001Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-31T16:00:02.001Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-31T16:00:02.001Z error prisma:error Error in PostgreSQL connection: Error { kind: Db, cause: Some(DbError { severity: "FATAL", parsed_severity: Some(Fatal), code: SqlState(E57P01), message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("postgres.c"), line: Some(3315), routine: Some("ProcessInterrupts") }) } langfuse-web-1 | 2025-08-31T16:00:02.139Z error Redis error connect ECONNREFUSED 172.20.0.4:6379 langfuse-worker-1 | 2025-09-01T14:18:31.604Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.81 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.27 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:18:31.604Z error ClickhouseWriter.flush traces (total) memory limit exceeded: would use 5.81 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.27 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:18:32.605Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.80 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.25 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:18:32.605Z error ClickhouseWriter.flush traces (total) memory limit exceeded: would use 5.80 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.25 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:18:33.604Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.85 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.25 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:18:33.604Z error ClickhouseWriter.flush traces (total) memory limit exceeded: would use 5.85 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.25 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:18:33.605Z error Max attempts reached for traces record. Dropping record. langfuse-worker-1 | 2025-09-01T14:18:34.608Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.88 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.21 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:18:34.608Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.88 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.21 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:18:35.608Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.96 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.25 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:18:35.608Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.96 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.25 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:18:36.608Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.94 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.28 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:18:36.608Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.94 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.28 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-web-1 | 2025-08-31T16:00:02.142Z error Redis error connect ECONNREFUSED 172.20.0.4:6379 langfuse-web-1 | 2025-08-31T16:00:02.144Z error Redis error connect ECONNREFUSED 172.20.0.4:6379 langfuse-worker-1 | 2025-09-01T14:18:36.608Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:18:36.608Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T14:18:38.608Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.93 GiB (attempt to allocate chunk of 4.59 MiB bytes), current RSS: 1.28 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:18:38.608Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.93 GiB (attempt to allocate chunk of 4.59 MiB bytes), current RSS: 1.28 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:18:39.610Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.99 GiB (attempt to allocate chunk of 4.59 MiB bytes), current RSS: 1.24 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:18:39.610Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.99 GiB (attempt to allocate chunk of 4.59 MiB bytes), current RSS: 1.24 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T14:18:40.610Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.93 GiB (attempt to allocate chunk of 4.59 MiB bytes), current RSS: 1.29 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T14:18:40.610Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.93 GiB (attempt to allocate chunk of 4.59 MiB bytes), current RSS: 1.29 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-web-1 | 2025-09-01T09:01:18.833Z error middleware intercepted error with code FORBIDDEN You cannot grant/edit a role higher than your own langfuse-worker-1 | 2025-09-01T14:18:40.610Z error Max attempts reached for observations record. Dropping record. langfuse-worker-1 | 2025-09-01T15:06:29.927Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.44 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.39 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T15:06:29.927Z error ClickhouseWriter.flush traces (total) memory limit exceeded: would use 5.44 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.39 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-web-1 | 2025-09-01T09:01:18.834Z error tRPC route failed on members.create: You cannot grant/edit a role higher than your own You cannot grant/edit a role higher than your own langfuse-web-1 | 2025-09-01T09:01:21.092Z error Missing environment variables for sending membership invitation email. langfuse-worker-1 | 2025-09-01T15:06:30.928Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.45 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.34 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T15:06:30.928Z error ClickhouseWriter.flush traces (total) memory limit exceeded: would use 5.45 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.34 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-web-1 | 2025-09-01T13:58:35.703Z error middleware intercepted error with code INTERNAL_SERVER_ERROR (total) memory limit exceeded: would use 5.43 GiB (attempt to allocate chunk of 0.00 B bytes), current RSS: 1.28 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing AggregatingTransform. langfuse-web-1 | 2025-09-01T13:58:35.704Z error tRPC route failed on traces.countAll: Internal error Internal error langfuse-web-1 | TRPCError: Internal error langfuse-web-1 | 2025-09-01T13:58:35.782Z error middleware intercepted error with code INTERNAL_SERVER_ERROR (total) memory limit exceeded: would use 5.45 GiB (attempt to allocate chunk of 4.25 MiB bytes), current RSS: 1.29 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: (while reading column input): (while reading from part /var/lib/clickhouse/store/e6f/e6fda190-65b2-41cf-89ee-90941c855b5c/202509_17311_17666_75/ in table default.observations (e6fda190-65b2-41cf-89ee-90941c855b5c) located on disk default of type local, from mark 2 with max_rows_to_read = 77, offset = 0): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). langfuse-web-1 | 2025-09-01T13:58:35.782Z error tRPC route failed on observations.byId: Internal error Internal error langfuse-web-1 | TRPCError: Internal error langfuse-worker-1 | 2025-09-01T15:06:31.929Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.45 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.37 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T15:06:31.929Z error ClickhouseWriter.flush traces (total) memory limit exceeded: would use 5.45 GiB (attempt to allocate chunk of 4.34 MiB bytes), current RSS: 1.37 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T15:06:31.929Z error Max attempts reached for traces record. Dropping record. langfuse-worker-1 | 2025-09-01T15:07:09.944Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.42 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.35 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T15:07:09.944Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.42 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.35 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T15:07:13.948Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.30 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T15:07:13.948Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.30 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T15:07:20.954Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.42 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.34 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T15:07:20.954Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.42 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.34 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T15:07:21.959Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 1.36 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T15:07:21.959Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.40 GiB (attempt to allocate chunk of 4.16 MiB bytes), current RSS: 1.36 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T15:08:05.974Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.44 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-web-1 | 2025-09-01T14:00:39.203Z error middleware intercepted error with code INTERNAL_SERVER_ERROR (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 0.00 B bytes), current RSS: 1.23 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing AggregatingTransform. langfuse-web-1 | 2025-09-01T14:00:39.203Z error tRPC route failed on traces.filterOptions: Internal error Internal error langfuse-web-1 | TRPCError: Internal error langfuse-worker-1 | 2025-09-01T15:08:05.974Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.41 GiB (attempt to allocate chunk of 4.60 MiB bytes), current RSS: 1.44 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T15:08:49.997Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.42 GiB (attempt to allocate chunk of 4.73 MiB bytes), current RSS: 1.35 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T15:08:49.998Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.42 GiB (attempt to allocate chunk of 4.73 MiB bytes), current RSS: 1.35 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-worker-1 | 2025-09-01T15:08:50.999Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.42 GiB (attempt to allocate chunk of 4.73 MiB bytes), current RSS: 1.30 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T15:08:50.999Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.42 GiB (attempt to allocate chunk of 4.73 MiB bytes), current RSS: 1.30 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | at parseError (/app/node_modules/.pnpm/@[email protected]/node_modules/@clickhouse/client-common/dist/error/error.js:38:16) langfuse-web-1 | 2025-09-01T14:00:39.274Z error Error executing query (total) memory limit exceeded: would use 5.42 GiB (attempt to allocate chunk of 0.00 B bytes), current RSS: 1.21 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing AggregatingTransform. langfuse-web-1 | 2025-09-01T14:00:39.274Z error middleware intercepted error with code INTERNAL_SERVER_ERROR Failed to execute query langfuse-worker-1 | 2025-09-01T15:08:51.998Z error ClickhouseWriter.writeToClickhouse Error: (total) memory limit exceeded: would use 5.43 GiB (attempt to allocate chunk of 4.73 MiB bytes), current RSS: 1.26 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert. langfuse-worker-1 | 2025-09-01T15:08:51.998Z error ClickhouseWriter.flush observations (total) memory limit exceeded: would use 5.43 GiB (attempt to allocate chunk of 4.73 MiB bytes), current RSS: 1.26 GiB, maximum: 5.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: While executing WaitForAsyncInsert.

secretxuan avatar Sep 02 '25 10:09 secretxuan

Hey @secretxuan , Can you confirm that you host machine has sufficient capacity to fulfill the minimum infrastructure requirements as outlined in https://langfuse.com/self-hosting/configuration/scaling?

Steffen911 avatar Sep 02 '25 12:09 Steffen911

Yes, I just git cloned your project on a brand new cvm, and then started the service using docker-compose up of docker engine. This cvm has nothing else on it. My entire physical machine has 32GB of virtual memory and a 16-core CPU.

secretxuan avatar Sep 03 '25 02:09 secretxuan

@secretxuan Could you attempt to add a memory reservation to the ClickHouse container that allocates 8GiB to it, e.g.

  clickhouse:
    image: docker.io/clickhouse/clickhouse-server
    restart: always
    user: "101:101"
    environment:
      CLICKHOUSE_DB: default
      CLICKHOUSE_USER: clickhouse
      CLICKHOUSE_PASSWORD: clickhouse # CHANGEME
    deploy:
      resources:
        reservations:
          memory: 8192M
    volumes:
      - langfuse_clickhouse_data:/var/lib/clickhouse
      - langfuse_clickhouse_logs:/var/log/clickhouse-server
    ports:
      - 127.0.0.1:8123:8123
      - 127.0.0.1:9000:9000
    healthcheck:
      test: wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1
      interval: 5s
      timeout: 5s
      retries: 10
      start_period: 1s

?

Steffen911 avatar Sep 03 '25 08:09 Steffen911