server
server copied to clipboard
Published
20 hours ago •
PrefectHQ
PrefectHQ
Backend Server is unable to start properly
Description
- Cleanly installed docker
- Created a new python 3.9 venv
- On starting prefect backend server getting the errors below
apollo_1 | Checking GraphQL service at http://graphql:4201/health ...
hasura_1 | {"type":"pg-client","timestamp":"2022-05-18T08:54:06.805+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
hasura_1 | {"type":"pg-client","timestamp":"2022-05-18T08:54:06.805+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(1)."}}
hasura_1 | {"type":"startup","timestamp":"2022-05-18T08:54:06.805+0000","level":"error","detail":{"kind":"catalog_migrate","info":{"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.22.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}}}
hasura_1 | {"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.22.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}
graphql_1 |
graphql_1 | Running Alembic migrations...
graphql_1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
graphql_1 | INFO [alembic.runtime.migration] Will assume transactional DDL.
graphql_1 | INFO [alembic.runtime.migration] Running upgrade -> 27811b58307b, Create extensions and initial settings
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 27811b58307b -> 72e2cd3e0469, Initial database tables migration
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 72e2cd3e0469 -> c4d792bdd05e, Add flow run idempotency key
graphql_1 | INFO [alembic.runtime.migration] Running upgrade c4d792bdd05e -> 3398e4807bfb, Add traversal functions
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 3398e4807bfb -> b9086bd4b962, Create message table
graphql_1 | INFO [alembic.runtime.migration] Running upgrade b9086bd4b962 -> c1f317aa658c, Remove state_id foreign keys
graphql_1 | INFO [alembic.runtime.migration] Running upgrade c1f317aa658c -> 6611fd0ccc73, Simplify run state update triggers
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 6611fd0ccc73 -> 70528cee0d2b, Add agent persistence
postgres_1 | The files belonging to this database system will be owned by user "postgres".
postgres_1 | This user must also own the server process.
postgres_1 |
postgres_1 | The database cluster will be initialized with locale "en_US.utf8".
postgres_1 | The default database encoding has accordingly been set to "UTF8".
postgres_1 | The default text search configuration will be set to "english".
postgres_1 |
postgres_1 | Data page checksums are disabled.
postgres_1 |
postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1 | creating subdirectories ... ok
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 70528cee0d2b -> 9cb7539b7363, Add index on agent_id
postgres_1 | selecting default max_connections ... 100
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 9cb7539b7363 -> e148cf9f1e5b, Add task run name
graphql_1 | INFO [alembic.runtime.migration] Running upgrade e148cf9f1e5b -> 850b76d44332, Add flow run config
postgres_1 | selecting default shared_buffers ... 128MB
postgres_1 | selecting default timezone ... Etc/UTC
postgres_1 | selecting dynamic shared memory implementation ... posix
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 850b76d44332 -> 24f10aeee83e, Add label column to flow runs
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 24f10aeee83e -> 3c87ad7e0b71, Add artifact api
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 3c87ad7e0b71 -> 57ac2cb01ac1, Add index for task run names
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 57ac2cb01ac1 -> 7ca57ea2fdff, Add run_config to flow runs and flow groups
postgres_1 | creating configuration files ... ok
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 7ca57ea2fdff -> 9116e81c6dc2, Add description to flow group table
postgres_1 | running bootstrap script ... ok
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 9116e81c6dc2 -> 459a61bedc9e, Improve run triggers to handle same-version states
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 459a61bedc9e -> a666a3f4e422, Add unique index for idempotency key
graphql_1 | INFO [alembic.runtime.migration] Running upgrade a666a3f4e422 -> ac5747fb571c, Add unique constraint on edge table for task IDs
postgres_1 | performing post-bootstrap initialization ... ok
postgres_1 | syncing data to disk ... ok
postgres_1 |
postgres_1 | Success. You can now start the database server using:
postgres_1 |
postgres_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1 |
postgres_1 |
postgres_1 | WARNING: enabling "trust" authentication for local connections
postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1 | --auth-local and --auth-host, the next time you run initdb.
postgres_1 | waiting for server to start....2022-05-18 08:54:06.567 UTC [48] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2022-05-18 08:54:06.608 UTC [49] LOG: database system was shut down at 2022-05-18 08:54:06 UTC
postgres_1 | 2022-05-18 08:54:06.622 UTC [48] LOG: database system is ready to accept connections
postgres_1 | done
postgres_1 | server started
postgres_1 | CREATE DATABASE
postgres_1 |
postgres_1 |
postgres_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1 |
postgres_1 | 2022-05-18 08:54:07.041 UTC [48] LOG: received fast shutdown request
postgres_1 | waiting for server to shut down....2022-05-18 08:54:07.042 UTC [48] LOG: aborting any active transactions
postgres_1 | 2022-05-18 08:54:07.045 UTC [48] LOG: background worker "logical replication launcher" (PID 55) exited with exit code 1
postgres_1 | 2022-05-18 08:54:07.054 UTC [50] LOG: shutting down
postgres_1 | 2022-05-18 08:54:07.068 UTC [48] LOG: database system is shut down
postgres_1 | done
postgres_1 | server stopped
postgres_1 |
postgres_1 | PostgreSQL init process complete; ready for start up.
postgres_1 |
postgres_1 | 2022-05-18 08:54:07.166 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres_1 | 2022-05-18 08:54:07.166 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres_1 | 2022-05-18 08:54:07.175 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2022-05-18 08:54:07.213 UTC [76] LOG: database system was shut down at 2022-05-18 08:54:07 UTC
postgres_1 | 2022-05-18 08:54:07.234 UTC [1] LOG: database system is ready to accept connections
ui_1 | Missing the PREFECT_SERVER__BASE_URL environment variable. Using default
ui_1 | 👾👾👾 UI running at localhost:8080 👾👾👾
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: using the "epoll" event method
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: nginx/1.20.2
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: OS: Linux 4.15.0-177-generic
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: getrlimit(RLIMIT_NOFILE): 1048576:1048576
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: start worker processes
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: start worker process 11
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: start worker process 12
apollo_1 | Checking GraphQL service at http://graphql:4201/health ...
apollo_1 | Checking GraphQL service at http://graphql:4201/health ...
towel_1 | {"severity": "ERROR", "name": "prefect-server.HasuraClient", "message": "Encountered internal API exception: All connection attempts failed", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 127, in try_connect\n stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1519, in connect_tcp\n local_addr=local_addr)\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n await self.sock_connect(sock, address)\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n return await fut\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('172.22.0.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 184, in connect_tcp\n raise OSError('All connection attempts failed') from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n response = await connection.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n stream = await self._connect(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n stream = await self._network_backend.connect_tcp(**kwargs)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 24, in connect_tcp\n host, port, timeout=timeout, local_address=local_address\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1833, in post\n extensions=extensions,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1506, in request\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1597, in send\n history=[],\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1624, in _send_handling_auth\n history=history,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1658, in _send_handling_redirects\n response = await self._send_single_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1695, in _send_single_request\n response = await transport.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed"}
towel_1 | {"severity": "ERROR", "name": "prefect-server.ZombieKiller", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 127, in try_connect\n stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1519, in connect_tcp\n local_addr=local_addr)\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n await self.sock_connect(sock, address)\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n return await fut\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('172.22.0.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 184, in connect_tcp\n raise OSError('All connection attempts failed') from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n response = await connection.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n stream = await self._connect(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n stream = await self._network_backend.connect_tcp(**kwargs)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 24, in connect_tcp\n host, port, timeout=timeout, local_address=local_address\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1833, in post\n extensions=extensions,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1506, in request\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1597, in send\n history=[],\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1624, in _send_handling_auth\n history=history,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1658, in _send_handling_redirects\n response = await self._send_single_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1695, in _send_single_request\n response = await transport.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n await self.run_once()\n File \"/prefect-server/src/prefect_server/services/towel/zombie_killer.py\", line 216, in run_once\n await self.reap_zombie_task_runs()\n File \"/prefect-server/src/prefect_server/services/towel/zombie_killer.py\", line 153, in reap_zombie_task_runs\n apply_schema=False,\n File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n as_box=not apply_schema,\n File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n as_box=as_box,\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 188, in __aexit__\n await self.gen.athrow(typ, value, traceback)\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 93, in reraise_as_api_error\n raise APIError() from exc\nprefect_server.utilities.exceptions.APIError: Unable to complete operation. An internal API error occurred."}
towel_1 | {"severity": "ERROR", "name": "prefect-server.HasuraClient", "message": "Encountered internal API exception: All connection attempts failed", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 127, in try_connect\n stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1519, in connect_tcp\n local_addr=local_addr)\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n await self.sock_connect(sock, address)\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n return await fut\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('172.22.0.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 184, in connect_tcp\n raise OSError('All connection attempts failed') from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n response = await connection.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n stream = await self._connect(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n stream = await self._network_backend.connect_tcp(**kwargs)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 24, in connect_tcp\n host, port, timeout=timeout, local_address=local_address\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1833, in post\n extensions=extensions,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1506, in request\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1597, in send\n history=[],\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1624, in _send_handling_auth\n history=history,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1658, in _send_handling_redirects\n response = await self._send_single_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1695, in _send_single_request\n response = await transport.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed"}
towel_1 | {"severity": "ERROR", "name": "prefect-server.Lazarus", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 127, in try_connect\n stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1519, in connect_tcp\n local_addr=local_addr)\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n await self.sock_connect(sock, address)\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n return await fut\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('172.22.0.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 184, in connect_tcp\n raise OSError('All connection attempts failed') from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n response = await connection.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n stream = await self._connect(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n stream = await self._network_backend.connect_tcp(**kwargs)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 24, in connect_tcp\n host, port, timeout=timeout, local_address=local_address\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1833, in post\n extensions=extensions,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1506, in request\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1597, in send\n history=[],\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1624, in _send_handling_auth\n history=history,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1658, in _send_handling_redirects\n response = await self._send_single_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1695, in _send_single_request\n response = await transport.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n await self.run_once()\n File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 37, in run_once\n return await self.reschedule_flow_runs()\n File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 89, in reschedule_flow_runs\n limit=500,\n File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n as_box=not apply_schema,\n File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n as_box=as_box,\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 188, in __aexit__\n await self.gen.athrow(typ, value, traceback)\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 93, in reraise_as_api_error\n raise APIError() from exc\nprefect_server.utilities.exceptions.APIError: Unable to complete operation. An internal API error occurred."}
towel_1 | {"severity": "ERROR", "name": "prefect-server.HasuraClient", "message": "Encountered internal API exception: All connection attempts failed", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 127, in try_connect\n stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1519, in connect_tcp\n local_addr=local_addr)\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n await self.sock_connect(sock, address)\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n return await fut\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('172.22.0.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 184, in connect_tcp\n raise OSError('All connection attempts failed') from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n response = await connection.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n stream = await self._connect(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n stream = await self._network_backend.connect_tcp(**kwargs)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 24, in connect_tcp\n host, port, timeout=timeout, local_address=local_address\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1833, in post\n extensions=extensions,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1506, in request\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1597, in send\n history=[],\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1624, in _send_handling_auth\n history=history,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1658, in _send_handling_redirects\n response = await self._send_single_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1695, in _send_single_request\n response = await transport.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed"}
towel_1 | {"severity": "ERROR", "name": "prefect-server.Scheduler", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 127, in try_connect\n stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1519, in connect_tcp\n local_addr=local_addr)\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n await self.sock_connect(sock, address)\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n return await fut\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('172.22.0.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 184, in connect_tcp\n raise OSError('All connection attempts failed') from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n response = await connection.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n stream = await self._connect(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n stream = await self._network_backend.connect_tcp(**kwargs)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 24, in connect_tcp\n host, port, timeout=timeout, local_address=local_address\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1833, in post\n extensions=extensions,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1506, in request\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1597, in send\n history=[],\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1624, in _send_handling_auth\n history=history,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1658, in _send_handling_redirects\n response = await self._send_single_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1695, in _send_single_request\n response = await transport.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n await self.run_once()\n File \"/prefect-server/src/prefect_server/services/towel/scheduler.py\", line 47, in run_once\n offset=500 * iterations,\n File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n as_box=not apply_schema,\n File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n as_box=as_box,\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 188, in __aexit__\n await self.gen.athrow(typ, value, traceback)\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 93, in reraise_as_api_error\n raise APIError() from exc\nprefect_server.utilities.exceptions.APIError: Unable to complete operation. An internal API error occurred."}
graphql_1 |
graphql_1 | Could not upgrade the database!
graphql_1 | Error: HTTPConnectionPool(host='hasura', port=3000): Max retries exceeded with url: /v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f826cc9e590>: Failed to establish a new connection: [Errno 111] Connection refused'))
apollo_1 | Checking GraphQL service at http://graphql:4201/health ...
graphql_1 |
graphql_1 | Running Alembic migrations...
graphql_1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
graphql_1 | INFO [alembic.runtime.migration] Will assume transactional DDL.
apollo_1 | Checking GraphQL service at http://graphql:4201/health ...
graphql_1 | Applied Hasura metadata from /prefect-server/services/hasura/migrations/metadata.yaml
graphql_1 |
graphql_1 | Database upgraded!
apollo_1 | Checking GraphQL service at http://graphql:4201/health ...
graphql_1 | {"severity": "INFO", "name": "prefect-server.GraphQL Server", "message": "Using uvicorn log level = 'debug'"}
graphql_1 | INFO: Started server process [8]
graphql_1 | INFO: Waiting for application startup.
graphql_1 | INFO: Application startup complete.
graphql_1 | INFO: Uvicorn running on http://0.0.0.0:4201 (Press CTRL+C to quit)
apollo_1 | Checking GraphQL service at http://graphql:4201/health ...
graphql_1 | INFO: 172.22.0.6:60896 - "GET /health HTTP/1.1" 200 OK
apollo_1 | {"status":"ok","version":"2022.04.14"}
apollo_1 | GraphQL service healthy!
apollo_1 |
apollo_1 | > @ serve /apollo
apollo_1 | > node dist/index.js
apollo_1 |
apollo_1 | Building schema...
graphql_1 | INFO: 172.22.0.6:60922 - "POST / HTTP/1.1" 200 OK
apollo_1 | Building schema complete!
apollo_1 | Server ready at http://0.0.0.0:4200 🚀 (version: 2022.04.14)
apollo_1 | Sending telemetry to Prefect Technologies, Inc.: {"source":"prefect_server","type":"startup","payload":{"id":"3b5b6298-f162-45d0-bf88-ea62e4eb573b","prefect_server_version":"2022.04.14","api_version":"0.2.0"}}
graphql_1 | INFO: 172.22.0.6:60944 - "POST / HTTP/1.1" 200 OK
graphql_1 | INFO: 172.22.0.6:60956 - "POST / HTTP/1.1" 200 OK
WELCOME TO
_____ _____ ______ ______ ______ _____ _______ _____ ______ _______ ________ _____
| __ \| __ \| ____| ____| ____/ ____|__ __| / ____| ____| __ \ \ / / ____| __ \
| |__) | |__) | |__ | |__ | |__ | | | | | (___ | |__ | |__) \ \ / /| |__ | |__) |
| ___/| _ /| __| | __| | __|| | | | \___ \| __| | _ / \ \/ / | __| | _ /
| | | | \ \| |____| | | |___| |____ | | ____) | |____| | \ \ \ / | |____| | \ \
|_| |_| \_\______|_| |______\_____| |_| |_____/|______|_| \_\ \/ |______|_| \_\
Visit http://localhost:8080 to get started, or check out the docs at https://docs.prefect.io
Reproduction
prefect backend server
prefect server start
Environment
This looks like it's running fine. The server will sometimes log errors while it is waiting for all of the services to spin up. Are your containers unhealthy at the end?
The UI does snip up but it is unable to get the data from previous runs and created projects and flow on doing a new clean install in an environment. Had first tried with python3.6, since there were some issues related towel service with that python version I recreated one with python3.9. I expected that the old data would persist but this seems a bit different had to re-register flows to new agents and projects. While there is a .prefect directory in the user folder.
On Wed, 18 May, 2022, 8:35 pm Michael Adkins, @.***> wrote:
This looks like it's running fine. The server will sometimes log errors while it is waiting for all of the services to spin up. Are your containers unhealthy at the end?
— Reply to this email directly, view it on GitHub https://github.com/PrefectHQ/server/issues/368#issuecomment-1130133797, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOC7FQZVLK5BNVHX3VGJO2LVKUBMFANCNFSM5WHYSUBQ . You are receiving this because you authored the thread.Message ID: @.***>
That .prefect folder is not used as a volume for the postgres database server uses.
You can find the relevant docs here how to persist data upon server restarts.
I did wipe out the older created containers
On Wed, 18 May, 2022, 8:35 pm Michael Adkins, @.***> wrote:
This looks like it's running fine. The server will sometimes log errors while it is waiting for all of the services to spin up. Are your containers unhealthy at the end?
— Reply to this email directly, view it on GitHub https://github.com/PrefectHQ/server/issues/368#issuecomment-1130133797, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOC7FQZVLK5BNVHX3VGJO2LVKUBMFANCNFSM5WHYSUBQ . You are receiving this because you authored the thread.Message ID: @.***>