prefect
prefect copied to clipboard
PrefectHQ
read_work_queue_by_name raises JSONDecodeError
First check
- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.
Bug summary
client: OrionClient
queue_name: "sync_assessments"
When running the following, raises JSONDecodeError:
work_queue = await client.read_work_queue_by_name(queue_name)
Referencing the Prefect source code and stepping through the execution with pdb:
response = await self._client.get(f"/work_queues/name/{name}") -> returns a 200 OK with response.is_success == True
and response.text looks like
'<!DOCTYPE html>\n<html lang="en">\n <head>\n <meta http-equiv="Cache-Control" content="max-age=0, must-revalidate"/>\n <meta http-equiv="Pragma" content="no-cache"/>\n <meta http-equiv="Expires" content="0" />\n <meta charset="UTF-8" />\n <link rel="apple-touch-icon" sizes="180x180" href="/ico/apple-touch-icon.png">\n \n <link rel="icon" id="favicon-32" type="image/png" sizes="32x32" href="/ico/favicon-32x32.png" media="(prefers-color-scheme:light)">\n <link rel="icon" id="favicon-16" type="image/png" sizes="16x16" href="/ico/favicon-16x16.png" media="(prefers-color-scheme:light)">\n\n <link rel="icon" id="favicon-32-dark" type="image/png" sizes="32x32" href="/ico/favicon-32x32-dark.png" media="(prefers-color-scheme:dark)">\n <link rel="icon" id="favicon-16-dark" type="image/png" sizes="16x16" href="/ico/favicon-16x16-dark.png" media="(prefers-color-scheme:dark)">\n\n <link rel="icon" href="/favicon.ico" media="(prefers-color-scheme:no-preference)">\n <link rel="icon" href="/favicon.ico" media="(prefers-color-scheme:light)">\n <link rel="icon" href="/favicon-dark.ico" media="(prefers-color-scheme:dark)">\n\n <link rel="manifest" href="/ico/site.webmanifest">\n <link rel="mask-icon" href="/ico/safari-pinned-tab.svg" color="#5bbad5">\n <meta name="msapplication-TileColor" content="#da532c">\n <meta name="theme-color" content="#ffffff">\n\n <script src="/registerSW.js"></script>\n\n <!-- https://stripe.com/docs/payments/quickstart -->\n <!-- Don’t include the script in a bundle or host it yourself. -->\n <script src="https://js.stripe.com/v3/"></script>\n\n <meta name="viewport" content="width=device-width, initial-scale=1.0" />\n <title>Prefect Cloud</title>\n <script type="module" crossorigin src="/index.50a5a629.js"></script>\n <link rel="stylesheet" href="/assets/index-18768f53.css">\n </head>\n <body>\n <div id="app"></div>\n \n </body>\n</html>\n'
which is definitely not valid JSON, so the return statement which looks like return WorkQueue.parse_obj(response.json()) is raising json.decoder.JSONDecodeError
Reproduction
Using `OrionClient` and any string as `queue_name`, make a call to `read_work_queue_by_name`
Error
This was reported by an engineer on our team with the following stack trace:
Traceback (most recent call last):
File "/Users/cwatts/Code/api/manage.py", line 31, in <module>
execute_from_command_line(sys.argv)
File "/Users/cwatts/.local/share/virtualenvs/api-mwSkRMLi/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
utility.execute()
File "/Users/cwatts/.local/share/virtualenvs/api-mwSkRMLi/lib/python3.9/site-packages/django/core/management/__init__.py", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/cwatts/.local/share/virtualenvs/api-mwSkRMLi/lib/python3.9/site-packages/django/core/management/base.py", line 402, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/cwatts/.local/share/virtualenvs/api-mwSkRMLi/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute
output = self.handle(*args, **options)
File "/Users/cwatts/Code/api/octave_core/management/commands/prefect.py", line 96, in handle
asyncio.run(self.register(flow_groups))
File "/usr/local/Cellar/[email protected]/3.9.17/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/Cellar/[email protected]/3.9.17/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/Users/cwatts/Code/api/octave_core/management/commands/prefect.py", line 120, in register
await flow_group.create_flow_deployments(tags=["agent:octave-core"])
File "/Users/cwatts/Code/api/octave_core/prefect/flows/__init__.py", line 196, in create_flow_deployments
deployment_ids.update(set(await flow_info.create_deployments(client, tags=deployment_tags.copy())))
File "/Users/cwatts/Code/api/octave_core/prefect/flows/flow_info.py", line 21, in create_deployments
deployment_id = await deploy.create_for_flow(
File "/Users/cwatts/Code/api/octave_core/prefect/flows/deployment.py", line 128, in create_for_flow
work_queue = await client.read_work_queue_by_name(self.queue)
File "/Users/cwatts/.local/share/virtualenvs/api-mwSkRMLi/lib/python3.9/site-packages/prefect/client/orchestration.py", line 920, in read_work_queue_by_name
return WorkQueue.parse_obj(response.json())
File "/Users/cwatts/.local/share/virtualenvs/api-mwSkRMLi/lib/python3.9/site-packages/httpx/_models.py", line 756, in json
return jsonlib.loads(self.text, **kwargs)
File "/usr/local/Cellar/[email protected]/3.9.17/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/local/Cellar/[email protected]/3.9.17/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/Cellar/[email protected]/3.9.17/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
### Versions
```Text
--- % prefect version
Version: 2.11.0
API version: 0.8.4
Python version: 3.9.16
Git commit: eeb9e219
Built: Thu, Jul 20, 2023 4:34 PM
OS/Arch: darwin/arm64
Profile: default
Server type: server
### Additional context
_No response_
I haven't reproduced this so first step for whoever picks it up is to check you can reproduce and add more details.
You should include /api in PREFECT_API_URL, e.g. export PREFECT_API_URL=http://127.0.0.1:4200/api
Facing the same issue when trying to use prefect behind nginx (via reverse proxy)
Following is my nginx conf:
upstream prefect_server {
server server:4200;
}
# now we declare our main server
server {
listen 80;
server_name _;
client_max_body_size 10M;
location /prefect/api {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
rewrite ^/prefect/api/?(.*)$ /api/$1 break;
proxy_pass http://prefect_server;
}
location /prefect/ {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
proxy_pass http://prefect_server;
}
}
Following is the compose.yaml:
services:
### Prefect Database
database:
image: postgres:alpine
restart: always
environment:
- POSTGRES_USER=${POSTGRES_USER}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
- POSTGRES_DB=${POSTGRES_DB}
networks:
- prefect_network
volumes:
- ./data/db:/var/lib/postgresql/data
### Nginx Reverse Proxy
nginx:
image: nginx:alpine
restart: always
ports:
- 8000:80
networks:
- ui_network
volumes:
- ./config/nginx/:/etc/nginx/conf.d/
depends_on:
- server
### Prefect Server API and UI
server:
image: prefecthq/prefect:3-python3.12
restart: always
volumes:
- prefect:/root/.prefect
entrypoint: ["/opt/prefect/entrypoint.sh", "prefect", "server", "start"]
environment:
- PREFECT_UI_URL=http://0.0.0.0:8000/prefect
- PREFECT_API_URL=/prefect/api
- PREFECT_UI_SERVE_BASE=/prefect
# If you want to access Prefect Server UI from anywhere other than the Docker host machine, you will need to change
# PREFECT_UI_URL and PREFECT_API_URL to match the external hostname/IP of the host machine. For example:
#- PREFECT_UI_URL=http://external-ip/prefect
#- PREFECT_API_URL=http://external-ip/prefect/api
- PREFECT_SERVER_API_HOST=${PREFECT_SERVER_API_HOST}
- PREFECT_API_DATABASE_CONNECTION_URL=${PREFECT_API_DATABASE_CONNECTION_URL}
# - EXTRA_PIP_PACKAGES=${EXTRA_PIP_PACKAGES}
depends_on:
- database
networks:
- ui_network
- prefect_network
### Auto-discovery and deployment service
deploy:
image: prefecthq/prefect:3-python3.12
restart: "no"
working_dir: "/root/flows"
volumes:
- "./flows:/root/flows"
- "./deployments:/root/deployments"
environment:
# - PREFECT_API_URL=${PREFECT_API_URL:-http://0.0.0.0:8000/prefect/api}
- PREFECT_API_URL=http://server:4200/prefect/api
- WORK_POOL_NAME=${WORK_POOL_NAME}
- WORK_POOL_TYPE=${WORK_POOL_TYPE}
depends_on:
- server
networks:
- prefect_network
entrypoint:
- bash
- -c
- |
echo 'Waiting for Prefect server to be ready...'
sleep 30
echo 'Creating work pool...'
prefect work-pool create $${WORK_POOL_NAME} --type $${WORK_POOL_TYPE} || echo 'Work pool already exists'
## Prefect Worker (Process-based for local execution)
worker:
image: prefecthq/prefect:3-python3.12
restart: always
entrypoint:
[
"/opt/prefect/entrypoint.sh",
"prefect",
"worker",
"start",
"--pool",
"${WORK_POOL_NAME}",
"--type",
"${WORK_POOL_TYPE}",
]
environment:
- PREFECT_API_URL=http://server:4200/prefect/api
# - PREFECT_API_URL=${PREFECT_API_URL:-http://0.0.0.0:8000/prefect/api}
# - EXTRA_PIP_PACKAGES=${EXTRA_PIP_PACKAGES}
networks:
- prefect_network
volumes:
- ./flows:/root/flows
depends_on:
- deploy
volumes:
prefect:
networks:
ui_network:
prefect_network:
You should include
/apiinPREFECT_API_URL, e.g.export PREFECT_API_URL=http://127.0.0.1:4200/api
THIS SHOULD BE PLACED AT THE TOP OF THE DOCUMENTATION. 😠
