pgvecto.rs
pgvecto.rs copied to clipboard
tensorchord
The default max_stack_depth parameter in the container image may be too small
I created a vector table with 1536 dimensions to store embedding values, and then the db kept reporting errors when I tried to create the vector index on the table. I found such errors in the log:
PANIC: ERRORDATA_STACK_SIZE exceeded
So I set up max_stack_depth=2MB and restarted the container, everything was fine.
Therefore, I think it's better to set a larger default value in the image, or remind this configuration in the document.
We have not seen this error before, is your query very complex?
No, just create an empty table and try to create index on the vector field.
btw, this problem did not occur when the container was first created, but after the database instance was started for a period of time, even if nothing was done after startup, and then found that the problem suddenly appeared after a day or two.
Update:
Get this error again: PANIC: ERRORDATA_STACK_SIZE exceeded, it seems that setting max_stack_depth does not fundamentally solve the problem.
Can you provide steps to reproduce the problem? We have not received any similar problem reports so far.
Did not do any special operation, suddenly abnormal, the service is started through docker.
The detail log:
[2024-04-18T08:01:19Z INFO service::utils::clean] Find directory "pg_vectors/indexes/101071".
[2024-04-18T08:01:19Z INFO service::utils::clean] Find directory "pg_vectors/indexes/101076".
[2024-04-18T08:01:19Z INFO service::utils::clean] Find directory "pg_vectors/indexes/101078".
[2024-04-18T08:01:19Z INFO service::utils::clean] Find directory "pg_vectors/indexes/101072".
[2024-04-18T08:01:19Z INFO service::utils::clean] Find directory "pg_vectors/indexes/101075".
[2024-04-18T08:01:19Z INFO service::utils::clean] Find directory "pg_vectors/indexes/101077".
[2024-04-18T08:01:19Z INFO service::utils::clean] Find directory "pg_vectors/indexes/101074".
[2024-04-18T08:01:19Z INFO service::utils::clean] Find directory "pg_vectors/indexes/101071/segments/94bd1901-a42c-40f4-8e6e-f6b9d7cb83b5".
[2024-04-18T08:01:19Z ERROR vectors::bgworker] Panickied. Info: PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }), location: Location { file: "crates/service/src/algorithms/quantization/scalar.rs", line: 83, col: 64 }, can_unwind: true, force_no_backtrace: false }. Backtrace: disabled backtrace.
2024-04-18 08:03:09.157 UTC [48] ERROR: pgvecto.rs: IPC connection is closed unexpected.
ADVICE: The error is raisen by background worker errors. Please check the full PostgreSQL log to get more information. Please read `https://docs.pgvecto.rs/admin/configuration.html`.
2024-04-18 08:03:09.157 UTC [48] STATEMENT: REINDEX DATABASE "test"
2024-04-18 08:03:10.172 UTC [48] ERROR: pgvecto.rs: IPC connection is closed unexpected.
ADVICE: The error is raisen by background worker errors. Please check the full PostgreSQL log to get more information. Please read `https://docs.pgvecto.rs/admin/configuration.html`.
2024-04-18 08:03:10.172 UTC [48] WARNING: AbortTransaction while in ABORT state
2024-04-18 08:03:25.173 UTC [48] ERROR: pgvecto.rs: IPC connection is closed unexpected.
ADVICE: The error is raisen by background worker errors. Please check the full PostgreSQL log to get more information. Please read `https://docs.pgvecto.rs/admin/configuration.html`.
2024-04-18 08:03:25.173 UTC [48] WARNING: AbortTransaction while in ABORT state
2024-04-18 08:03:40.173 UTC [48] ERROR: pgvecto.rs: IPC connection is closed unexpected.
ADVICE: The error is raisen by background worker errors. Please check the full PostgreSQL log to get more information. Please read `https://docs.pgvecto.rs/admin/configuration.html`.
2024-04-18 08:03:40.174 UTC [48] WARNING: AbortTransaction while in ABORT state
2024-04-18 08:03:55.174 UTC [48] ERROR: pgvecto.rs: IPC connection is closed unexpected.
ADVICE: The error is raisen by background worker errors. Please check the full PostgreSQL log to get more information. Please read `https://docs.pgvecto.rs/admin/configuration.html`.
2024-04-18 08:03:55.174 UTC [48] PANIC: ERRORDATA_STACK_SIZE exceeded
2024-04-18 08:03:58.679 UTC [1] LOG: server process (PID 48) was terminated by signal 6: Aborted
2024-04-18 08:03:58.679 UTC [1] DETAIL: Failed process was running: REINDEX DATABASE "test"
2024-04-18 08:03:58.679 UTC [1] LOG: terminating any other active server processes
2024-04-18 08:03:58.681 UTC [1] LOG: all server processes terminated; reinitializing
2024-04-18 08:03:58.827 UTC [62] LOG: database system was interrupted; last known up at 2024-04-18 08:01:18 UTC
2024-04-18 08:03:58.830 UTC [65] FATAL: the database system is in recovery mode
2024-04-18 08:03:59.195 UTC [62] LOG: database system was not properly shut down; automatic recovery in progress
2024-04-18 08:03:59.241 UTC [62] LOG: redo starts at 0/D3C1370
2024-04-18 08:03:59.243 UTC [62] LOG: invalid record length at 0/D3C9B20: expected at least 24, got 0
2024-04-18 08:03:59.243 UTC [62] LOG: redo done at 0/D3C9AE8 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
2024-04-18 08:03:59.314 UTC [63] LOG: checkpoint starting: end-of-recovery immediate wait
2024-04-18 08:03:59.675 UTC [63] LOG: checkpoint complete: wrote 7 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.037 s, sync=0.154 s, total=0.401 s; sync files=6, longest=0.115 s, average=0.026 s; distance=34 kB, estimate=34 kB; lsn=0/D3C9B20, redo lsn=0/D3C9B20
2024-04-18 08:03:59.724 UTC [1] LOG: database system is ready to accept connections
How long has this image been running? How many vectors do you have? Does restart make it work well?
The container has been running for several days, but restart does not make it work, I have already tried.
There are only about 10 tables in the database, hundreds rows in each table, vectors have 1536 dimensions.
At present, you can also query on the table, but any write operation will report an error, including reindex, vacuum...
I notice the this line in the error log:
[2024-04-18T08:01:19Z ERROR vectors::bgworker] Panickied. Info: PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }), location: Location { file: "crates/service/src/algorithms/quantization/scalar.rs", line: 83, col: 64 }, can_unwind: true, force_no_backtrace: false }. Backtrace: disabled backtrace.
Is this relative? I created all tables with the option WITH ( options = "[indexing.hnsw.quantization.scalar]" )
Sorry to hear that. Currently you may need to manually delete all the files under pgdata/pgvecto_rs and run REINDEX for the indexes. The problem might be related to scalar quantization.
Delete all index under pg_vectors/indexes and restart the container and reindex database seems solve the problem.
However, I'm not sure if it won't be a problem without using scalar quantization. I'll update it. Anyway, there might be a bug here.
The error you met may due to code: 2, kind: NotFound, message: "No such file or directory", so that too many error accumulated. Do you meet the error after inserting the new vectors?