trypromptly
connection to server at "postgres" (172.18.0.3), port 5432 failed: FATAL: password authentication failed for user "llmstack"
On Ubuntu 24.04 using llmstack 0.2.5:
llmstack
Container llmstack-weaviate-1 Started
Container llmstack-redis-1 Started
Container llmstack-postgres-1 Started
Container llmstack-api-1 Starting
Container llmstack-rqworker-1 Starting
Container llmstack-runner-1 Started
Container llmstack-api-1 Started
Container llmstack-app-1 Starting
Container llmstack-rqworker-1 Started
Container llmstack-app-1 Startedartedg
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Failed to connect to LLMStack server. Exiting...
redis-1 | 1:C 26 Sep 2024 07:37:01.830 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis-1 | 1:C 26 Sep 2024 07:37:01.830 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis-1 | 1:C 26 Sep 2024 07:37:01.830 * Redis version=7.4.0, bits=64, commit=00000000, modified=0, pid=1, just started
redis-1 | 1:C 26 Sep 2024 07:37:01.830 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis-1 | 1:M 26 Sep 2024 07:37:01.831 * monotonic clock: POSIX clock_gettime
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * Running mode=standalone, port=6379.
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * Server initialized
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * Loading RDB produced by version 7.4.0
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * RDB age 307 seconds
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * RDB memory usage when created 1.30 Mb
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * Done loading RDB, keys loaded: 3, keys expired: 3.
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * DB loaded from disk: 0.000 seconds
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * Ready to accept connections tcp
rqworker-1 | postgres:5432 - accepting connections
rqworker-1 | Postgres is up - continuing
rqworker-1 | Starting RQ worker
rqworker-1 | /usr/local/lib/python3.11/site-packages/langrocks/common/models/tools_pb2_grpc.py:23: RuntimeWarning: The grpc package installed is at version 1.64.1, but the generated code in tools_pb2_grpc.py depends on grpcio>=1.65.1. Please upgrade your grpc module to grpcio>=1.65.1 or downgrade your generated code using grpcio-tools<=1.64.1. This warning will become an error in 1.66.0, scheduled for release on August 6, 2024.
rqworker-1 | warnings.warn(
rqworker-1 | /usr/local/lib/python3.11/site-packages/pydantic/_internal/_fields.py:201: UserWarning: Field name "schema" in "PandasStore" shadows an attribute in parent "BaseDestination"
rqworker-1 | warnings.warn(
rqworker-1 | /usr/local/lib/python3.11/site-packages/pydantic/_internal/_config.py:291: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
rqworker-1 | warnings.warn(DEPRECATION_MESSAGE, DeprecationWarning)
rqworker-1 | /usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:273: PydanticDeprecatedSince20: json_encoders is deprecated. See https://docs.pydantic.dev/2.8/concepts/serialization/#custom-serializers for alternatives. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
rqworker-1 | warnings.warn(
rqworker-1 | /usr/local/lib/python3.11/site-packages/pydantic/fields.py:814: PydanticDeprecatedSince20: Using extra keyword arguments on Field is deprecated and will be removed. Use json_schema_extra instead. (Extra keys: 'maximum', 'minimum'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
rqworker-1 | warn(
rqworker-1 | /usr/local/lib/python3.11/site-packages/pydantic/fields.py:783: PydanticDeprecatedSince20: min_items is deprecated and will be removed, use min_length instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
api-1 | postgres:5432 - accepting connections
rqworker-1 | warn('min_items is deprecated and will be removed, use min_length instead', DeprecationWarning)
app-1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
api-1 | Starting API server
api-1 | Postgres is up - continuing
app-1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
runner-1 | Starting redis server
api-1 | /usr/local/lib/python3.11/site-packages/langrocks/common/models/tools_pb2_grpc.py:23: RuntimeWarning: The grpc package installed is at version 1.64.1, but the generated code in tools_pb2_grpc.py depends on grpcio>=1.65.1. Please upgrade your grpc module to grpcio>=1.65.1 or downgrade your generated code using grpcio-tools<=1.64.1. This warning will become an error in 1.66.0, scheduled for release on August 6, 2024.
app-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
api-1 | warnings.warn(
api-1 | /usr/local/lib/python3.11/site-packages/pydantic/_internal/_fields.py:201: UserWarning: Field name "schema" in "PandasStore" shadows an attribute in parent "BaseDestination"
api-1 | warnings.warn(
rqworker-1 | /usr/local/lib/python3.11/site-packages/pydantic/fields.py:814: PydanticDeprecatedSince20: Using extra keyword arguments on Field is deprecated and will be removed. Use json_schema_extra instead. (Extra keys: 'example'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
api-1 | /usr/local/lib/python3.11/site-packages/pydantic/_internal/_config.py:291: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
app-1 | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
runner-1 | Starting playwright server on port 50053
app-1 | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
runner-1 | Listening on ws://localhost:50053/
app-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
runner-1 | INFO:langrocks.common.display:Created display: 99
app-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
runner-1 | INFO:langrocks.common.display:Created display: 100
app-1 | /docker-entrypoint.sh: Configuration complete; ready for start up
weaviate-1 | {"action":"startup","default_vectorizer_module":"text2vec-openai","level":"info","msg":"the default vectorizer modules is set to "text2vec-openai", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-09-26T07:37:01Z"}
weaviate-1 | {"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to "true"","time":"2024-09-26T07:37:01Z"}
weaviate-1 | {"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-09-26T07:37:01Z"}
weaviate-1 | {"level":"info","msg":"open cluster service","servers":{"weaviate-node":8300},"time":"2024-09-26T07:37:01Z"}
postgres-1 |
runner-1 | INFO:langrocks.common.display:Created display: 101
postgres-1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres-1 |
postgres-1 | 2024-09-26 07:37:01.920 UTC [1] LOG: starting PostgreSQL 16.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20240309) 13.2.1 20240309, 64-bit
postgres-1 | 2024-09-26 07:37:01.920 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres-1 | 2024-09-26 07:37:01.920 UTC [1] LOG: listening on IPv6 address "::", port 5432
runner-1 | INFO:langrocks.common.display:Created display: 102
weaviate-1 | {"address":"172.18.0.2:8301","level":"info","msg":"starting cloud rpc server ...","time":"2024-09-26T07:37:01Z"}
runner-1 | INFO:langrocks.common.display:Created display: 103
rqworker-1 | warn(
rqworker-1 | /code/llmstack/common/blocks/llm/openai.py:781: PydanticDeprecatedSince20: pydantic.config.Extra is deprecated, use literal values instead (e.g. extra='allow'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
runner-1 | WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
weaviate-1 | {"level":"info","msg":"starting raft sub-system ...","time":"2024-09-26T07:37:01Z"}
app-1 | 2024/09/26 07:37:02 [notice] 1#1: using the "epoll" event method
api-1 | warnings.warn(DEPRECATION_MESSAGE, DeprecationWarning)
app-1 | 2024/09/26 07:37:02 [notice] 1#1: nginx/1.26.1
weaviate-1 | {"address":"172.18.0.2:8300","level":"info","msg":"tcp transport","tcpMaxPool":3,"tcpTimeout":10000000000,"time":"2024-09-26T07:37:01Z"}
weaviate-1 | {"level":"info","msg":"loading local db","time":"2024-09-26T07:37:01Z"}
weaviate-1 | {"level":"info","msg":"database has been successfully loaded","n":0,"time":"2024-09-26T07:37:01Z"}
weaviate-1 | {"level":"info","metadata_only_voters":false,"msg":"construct a new raft node","name":"weaviate-node","time":"2024-09-26T07:37:01Z"}
weaviate-1 | {"action":"raft","index":1,"level":"info","msg":"raft initial configuration","servers":"[[{Suffrage:Voter ID:weaviate-node Address:172.18.0.2:8300}]]","time":"2024-09-26T07:37:02Z"}
weaviate-1 | {"last_snapshot_index":0,"last_store_applied_index":0,"last_store_log_applied_index":0,"level":"info","msg":"raft node constructed","raft_applied_index":0,"raft_last_index":18,"time":"2024-09-26T07:37:02Z"}
weaviate-1 | {"action":"raft","follower":{},"leader-address":"","leader-id":"","level":"info","msg":"raft entering follower state","time":"2024-09-26T07:37:02Z"}
weaviate-1 | {"action":"bootstrap","error":"could not join a cluster from [172.18.0.2:8300]","level":"warning","msg":"failed to join cluster, will notify next if voter","servers":["172.18.0.2:8300"],"time":"2024-09-26T07:37:03Z","voter":true}
app-1 | 2024/09/26 07:37:02 [notice] 1#1: built by gcc 13.2.1 20231014 (Alpine 13.2.1_git20231014)
weaviate-1 | {"action":"bootstrap","candidates":[{"Suffrage":0,"ID":"weaviate-node","Address":"172.18.0.2:8300"}],"level":"info","msg":"starting cluster bootstrapping","time":"2024-09-26T07:37:03Z"}
weaviate-1 | {"action":"bootstrap","error":"bootstrap only works on new clusters","level":"error","msg":"could not bootstrapping cluster","time":"2024-09-26T07:37:03Z"}
app-1 | 2024/09/26 07:37:02 [notice] 1#1: OS: Linux 6.8.0-45-generic
rqworker-1 | extra = Extra.forbid
app-1 | 2024/09/26 07:37:02 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker processes
rqworker-1 | System check identified some issues:
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 20
rqworker-1 |
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 21
postgres-1 | 2024-09-26 07:37:01.920 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
weaviate-1 | {"action":"bootstrap","level":"info","msg":"notified peers this node is ready to join as voter","servers":["172.18.0.2:8300"],"time":"2024-09-26T07:37:03Z"}
weaviate-1 | {"action":"raft","last-leader-addr":"","last-leader-id":"","level":"warning","msg":"raft heartbeat timeout reached, starting election","time":"2024-09-26T07:37:03Z"}
weaviate-1 | {"action":"raft","level":"info","msg":"raft entering candidate state","node":{},"term":19,"time":"2024-09-26T07:37:03Z"}
postgres-1 | 2024-09-26 07:37:01.924 UTC [29] LOG: database system was shut down at 2024-09-26 07:31:54 UTC
rqworker-1 | WARNINGS:
runner-1 | I0000 00:00:1727336222.178037 1 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
api-1 | /usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:273: PydanticDeprecatedSince20: json_encoders is deprecated. See https://docs.pydantic.dev/2.8/concepts/serialization/#custom-serializers for alternatives. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
api-1 | warnings.warn(
api-1 | Traceback (most recent call last):
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 22
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection
postgres-1 | 2024-09-26 07:37:01.932 UTC [1] LOG: database system is ready to accept connections
runner-1 | INFO:langrocks.app.server:Server running at http://[::]:50051
weaviate-1 | {"action":"raft","level":"info","msg":"raft election won","tally":1,"term":19,"time":"2024-09-26T07:37:03Z"}
postgres-1 | 2024-09-26 07:37:07.158 UTC [35] FATAL: password authentication failed for user "llmstack"
rqworker-1 | apiabstractor.Feedback.response_quality: (fields.W122) 'max_length' is ignored when used with SmallIntegerField.
weaviate-1 | {"action":"raft","leader":{},"level":"info","msg":"raft entering leader state","time":"2024-09-26T07:37:03Z"}
rqworker-1 | HINT: Remove 'max_length' from field
rqworker-1 | sheets.PromptlySheetRunEntry.data: (fields.E010) JSONField default should be a callable instead of an instance so that it's not shared between all field instances.
rqworker-1 | HINT: Use a callable instead, e.g., use dict instead of {}.
weaviate-1 | {"level":"warning","msg":"Multiple vector spaces are present, GraphQL Explore and REST API list objects endpoint module include params has been disabled as a result.","time":"2024-09-26T07:37:03Z"}
api-1 | self.connect()
api-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
api-1 | return func(*args, **kwargs)
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 23
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 24
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 25
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 26
rqworker-1 | /usr/local/lib/python3.11/site-packages/rq/connections.py:35: DeprecationWarning: The Connection context manager is deprecated. Use the connection parameter instead.
runner-1 | WebSocket server settings:
rqworker-1 | warnings.warn(
runner-1 | - Listen on :50052
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 27
runner-1 | - Web server. Web root: /usr/share/www/html
runner-1 | - No SSL/TLS support (no cert file)
runner-1 | - proxying from :50052 to targets generated by TokenRedis
rqworker-1 | /usr/local/lib/python3.11/site-packages/rq/connections.py:58: DeprecationWarning: The push_connection function is deprecated. Pass the connection explicitly instead.
rqworker-1 | warnings.warn(
postgres-1 | 2024-09-26 07:37:07.158 UTC [35] DETAIL: Password does not match for user "llmstack".
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 28
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 29
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 30
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 31
postgres-1 | Connection matched file "/var/lib/postgresql/data/pgdata/pg_hba.conf" line 128: "host all all all password"
weaviate-1 | {"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-09-26T07:37:03Z"}
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 32
weaviate-1 | {"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-09-26T07:37:03Z"}
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 33
weaviate-1 | {"address":"172.18.0.2:8300","level":"info","msg":"current Leader","time":"2024-09-26T07:37:04Z"}
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 34
weaviate-1 | {"level":"info","msg":"starting migration from old schema","time":"2024-09-26T07:37:04Z"}
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 35
weaviate-1 | {"level":"info","msg":"legacy schema is empty, nothing to migrate","time":"2024-09-26T07:37:04Z"}
weaviate-1 | {"level":"info","msg":"migration from the old schema has been successfully completed","time":"2024-09-26T07:37:04Z"}
postgres-1 | 2024-09-26 07:37:07.285 UTC [36] FATAL: password authentication failed for user "llmstack"
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 36
weaviate-1 | {"action":"bootstrap","level":"info","msg":"node reporting ready, node has probably recovered cluster from raft config. Exiting bootstrap process","time":"2024-09-26T07:37:04Z"}
api-1 | ^^^^^^^^^^^^^^^^^^^^^
postgres-1 | 2024-09-26 07:37:07.285 UTC [36] DETAIL: Password does not match for user "llmstack".
weaviate-1 | {"action":"telemetry_push","level":"info","msg":"telemetry started","payload":"\u0026{MachineID:ca9b4150-2194-4e11-b54a-82c2c41908f2 Type:INIT Version:1.25.7 NumObjects:0 OS:linux Arch:amd64 UsedModules:[]}","time":"2024-09-26T07:37:04Z"}
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 256, in connect
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 37
api-1 | self.connection = self.get_new_connection(conn_params)
api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
postgres-1 | Connection matched file "/var/lib/postgresql/data/pgdata/pg_hba.conf" line 128: "host all all all password"
api-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
api-1 | return func(*args, **kwargs)
postgres-1 | 2024-09-26 07:37:07.694 UTC [37] FATAL: password authentication failed for user "llmstack"
api-1 | ^^^^^^^^^^^^^^^^^^^^^
postgres-1 | 2024-09-26 07:37:07.694 UTC [37] DETAIL: Password does not match for user "llmstack".
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 277, in get_new_connection
postgres-1 | Connection matched file "/var/lib/postgresql/data/pgdata/pg_hba.conf" line 128: "host all all all password"
api-1 | connection = self.Database.connect(**conn_params)
api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/psycopg2/init.py", line 122, in connect
api-1 | conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-1 | psycopg2.OperationalError: connection to server at "postgres" (172.18.0.3), port 5432 failed: FATAL: password authentication failed for user "llmstack"
api-1 |
api-1 |
api-1 | The above exception was the direct cause of the following exception:
api-1 |
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 38
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 39
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 40
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 41
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 42
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 43
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 44
api-1 | Traceback (most recent call last):
api-1 | File "/code/manage.py", line 22, in
HI all , I have exactly the same issue on ubuntu 22.04.03 and 22.04.05, is there a problem with ubuntu ? If yes, which distribution is recommanded ?
Hi all, I also had similar issue (the error logs are lengthy). But I saw the last bit of the logs, which indicated authentication failed for postgres db. Then I edited a line from docker-compose.yml to "POSTGRES_HOST_AUTH_METHOD: "TRUST"". This docker-compose.yml was under this path "~/.local/lib/python3.11/site-packages/llmstack"
After that it didn't show any postgres auth issue. But there were further issues from mine. Hope this works for your case.
