Flowminder
Flowclient - API returned with status code: 500 and status 'Unknown status'
Describe the bug get_result method raises errors.
Product Flowclient
Version latest version of the master branch
To Reproduce
I set up Flowkit following quick install guide.
- Run: bash <(curl -s https://raw.githubusercontent.com/Flowminder/FlowKit/master/quick_start.sh) examples
- Generate a token using Flowauth
- Open one of the worked example notebooks
- Enter token and run cells
- Error comes with the use of "get_result" method.
Expected behavior
/opt/conda/lib/python3.9/site-packages/flowclient/client.py in get_result(connection, query_spec, disable_progress) 417 418 """ --> 419 return get_result_by_query_id( 420 connection=connection, 421 query_id=run_query(connection=connection, query_spec=query_spec),
/opt/conda/lib/python3.9/site-packages/flowclient/client.py in get_result_by_query_id(connection, query_id, poll_interval, disable_progress) 357 disable_progress=disable_progress, 358 ) --> 359 return get_json_dataframe(connection=connection, location=result_endpoint) 360 361
/opt/conda/lib/python3.9/site-packages/flowclient/client.py in get_json_dataframe(connection, location) 261 """ 262 --> 263 response = connection.get_url(route=location) 264 if response.status_code != 200: 265 try:
/opt/conda/lib/python3.9/site-packages/flowclient/connection.py in get_url(self, route, data) 149 except (ValueError, KeyError): 150 status = "Unknown status" --> 151 raise FlowclientConnectionError( 152 f"Something went wrong: {error}. API returned with status code: {response.status_code} and status '{status}'" 153 )
FlowclientConnectionError: Something went wrong: Unknown error. API returned with status code: 500 and status 'Unknown status'
Additional context
I also ran the FlowClient notebook files by installing FlowKit after running set -a && . development_environment && set +a followed by make up in the root directory. This time I am getting an error with the status code 400, error message: "Could not get reply for message". The error tracebacks (line 207 in connection.py <- line 523 in client.py <- line 421 in client.py)
Hi @bilgecag. Could you please share the outputs from running docker logs flowmachine and docker logs flowapi on your host machine after you encounter the error? A 500 code indicates either flowapi or flowmachine has encountered an error, so the logs should contain a more detailed error message.
Hi @jc-harrison , thank you for the answer. It seems like a flowapi error, after I ran docker logs flowapi I got the following output:
[...] {"event":"Closing connection to FlowMachine server\u2026","request":{"request_id":"e1bf7747-c199-479a-a097-0f67e43d8535","path":"/api/0/get/5d0fb9692963dd6ac2fdb10fe43e260c","src_ip":"172.18.0.5","user":"None"},"logger":"flowapi.debug","level":"debug","timestamp":"2022-01-14T15:05:38.028069Z"} {"event":"Closed socket.","request":{"request_id":"e1bf7747-c199-479a-a097-0f67e43d8535","path":"/api/0/get/5d0fb9692963dd6ac2fdb10fe43e260c","src_ip":"172.18.0.5","user":"None"},"logger":"flowapi.debug","level":"debug","timestamp":"2022-01-14T15:05:38.028240Z"} [2022-01-14 15:05:38 +0000] [12] [ERROR] Error in ASGI Framework Traceback (most recent call last): File "/root/.local/share/virtualenvs/flowapi-lc0QOo1U/lib/python3.8/site-packages/hypercorn/asyncio/task_group.py", line 21, in _handle await invoke_asgi(app, scope, receive, send) File "/root/.local/share/virtualenvs/flowapi-lc0QOo1U/lib/python3.8/site-packages/hypercorn/utils.py", line 247, in invoke_asgi await app(scope, receive, send) File "/root/.local/share/virtualenvs/flowapi-lc0QOo1U/lib/python3.8/site-packages/quart/app.py", line 1741, in call await self.asgi_app(scope, receive, send) File "/root/.local/share/virtualenvs/flowapi-lc0QOo1U/lib/python3.8/site-packages/quart/app.py", line 1767, in asgi_app await asgi_handler(receive, send) File "/root/.local/share/virtualenvs/flowapi-lc0QOo1U/lib/python3.8/site-packages/quart/asgi.py", line 51, in call _raise_exceptions(done) File "/root/.local/share/virtualenvs/flowapi-lc0QOo1U/lib/python3.8/site-packages/quart/asgi.py", line 353, in _raise_exceptions raise task.exception() File "/root/.local/share/virtualenvs/flowapi-lc0QOo1U/lib/python3.8/site-packages/quart/asgi.py", line 102, in handle_request await asyncio.wait_for(self._send_response(send, response), timeout=timeout) File "/usr/local/lib/python3.8/asyncio/tasks.py", line 494, in wait_for return fut.result() File "/root/.local/share/virtualenvs/flowapi-lc0QOo1U/lib/python3.8/site-packages/quart/asgi.py", line 131, in _send_response async for data in response_body: File "/root/.local/share/virtualenvs/flowapi-lc0QOo1U/lib/python3.8/site-packages/quart/helpers.py", line 259, in generator async for data in func(*args, **kwargs): File "/FlowKit-1.16.0+69.gd7ee3c06e/flowapi/flowapi/stream_results.py", line 33, in stream_result_as_json db_conn_pool = current_app.db_conn_pool AttributeError: 'Quart' object has no attribute 'db_conn_pool'
I do not see any errors in the logs of flowmachine.
I can confirm I am able to reproduce this error. I have not yet identified the underlying issue.
The issue here is that if the flowapi container starts up before flowdb is ready to accept connections, then flowapi's attempt to get a flowdb connection will fail, but this won't cause any downstream failures until it attempts to use the db connection to get a query result. Some possible solutions for us could be:
- If flowapi fails to create a db connection pool, retry until it succeeds
- If flowapi fails to create a db connection poll, restart the container (and repeat until it succeeds) - i.e. similar to the first suggestion but bring the entire service down on failure
- If connection pool doesn't exist when flowapi tries to use it, create a new one
@bilgecag: you should be able to resolve this issue by restarting the flowapi container while leaving the other containers running - docker restart flowapi
Dear @jc-harrison thank you for your answer. Restarting flowapi indeed solves the problem.