spec-first
Regression: Worse POST body handling compared to 2.14.0
Description
In a project we test for 415 status code for one of the endpoints. We are trying to update Python and dependencies to newer versions but some test cases fail in a way that seems like a regression compared to old version. Main issue is handling/decoding non JSON request body for data which in schema is declared as JSON.
Expected behaviour
Both test cases succeed and for both response.json() should return:
{"detail": "Invalid Content-type (), expected JSON data", "status": 415, "title": "Unsupported Media Type", "type": "about:blank"}
Actual behaviour
Test case with bytes respons with
{"type": "about:blank", "title": "Internal Server Error", "detail": "The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.", "status": 500}
and case with base64 string respons with
{"type": "about:blank", "title": "Bad Request", "detail": "Expecting value: line 1 column 1 (char 0)", "status": 400}
At least it's 400 instead 500 but 415 was better and description is worse than before.
Traceback for bytes case:
ERROR connexion.middleware.exceptions:exceptions.py:97 UnicodeDecodeError('utf-8', b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00', 0, 1, 'invalid start byte')
Traceback (most recent call last):
File "/hidden_path/lib/python3.12/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
await app(scope, receive, sender)
File "/hidden_path/lib/python3.12/site-packages/connexion/middleware/swagger_ui.py", line 220, in __call__
await self.router(scope, receive, send)
File "/hidden_path/lib/python3.12/site-packages/starlette/routing.py", line 746, in __call__
await route.handle(scope, receive, send)
File "/hidden_path/lib/python3.12/site-packages/starlette/routing.py", line 459, in handle
await self.app(scope, receive, send)
File "/hidden_path/lib/python3.12/site-packages/starlette/routing.py", line 775, in __call__
await self.default(scope, receive, send)
File "/hidden_path/lib/python3.12/site-packages/connexion/middleware/swagger_ui.py", line 233, in default_fn
await self.app(original_scope, receive, send)
File "/hidden_path/lib/python3.12/site-packages/connexion/middleware/routing.py", line 153, in __call__
await self.router(scope, receive, send)
File "/hidden_path/lib/python3.12/site-packages/starlette/routing.py", line 746, in __call__
await route.handle(scope, receive, send)
File "/hidden_path/lib/python3.12/site-packages/starlette/routing.py", line 459, in handle
await self.app(scope, receive, send)
File "/hidden_path/lib/python3.12/site-packages/starlette/routing.py", line 746, in __call__
await route.handle(scope, receive, send)
File "/hidden_path/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/hidden_path/lib/python3.12/site-packages/connexion/middleware/routing.py", line 47, in __call__
await self.next_app(original_scope, receive, send)
File "/hidden_path/lib/python3.12/site-packages/connexion/middleware/abstract.py", line 264, in __call__
return await operation(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/hidden_path/lib/python3.12/site-packages/connexion/middleware/security.py", line 106, in __call__
await self.next_app(scope, receive, send)
File "/hidden_path/lib/python3.12/site-packages/connexion/middleware/abstract.py", line 264, in __call__
return await operation(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/hidden_path/lib/python3.12/site-packages/connexion/middleware/request_validation.py", line 140, in __call__
receive = await validator.wrap_receive(receive, scope=scope)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/hidden_path/lib/python3.12/site-packages/connexion/validators/abstract.py", line 142, in wrap_receive
body = await self._parse(stream(), scope=scope)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/hidden_path/lib/python3.12/site-packages/connexion/validators/json.py", line 54, in _parse
body = bytes_body.decode(self._encoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte
It is clear here that connexion tries to decode body with assumption that it's possible to decode it as json and additionally doesn't handle thrown UnicodeDecodeError resulting in 500 status code instead something more meaningful. I can't even handle it myself because it happens before endpoint function is run.
Steps to reproduce
Files
.
├── api.yaml
├── app.py
├── parse.py
└── test_parse.py
spec
# api.yaml
openapi: 3.0.0
info:
title: title
version: 1.0.0
paths:
/path:
post:
description: Parse html
operationId: parse.post
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
html:
type: string
format: html
description: HTML to parse.
url:
type: string
format: url
description: URL of webpage from which HTML was taken.
headers:
type: object
description: Headers of webpage from which HTML was taken.
required:
- html
- url
responses:
200:
description: Returns HTML text
content:
application/json:
schema:
type: object
properties:
text:
type: string
endpoint func (kinda doesn't matter because it doesn't even go inside the funcion):
# parse.py
from typing import Any, cast, Dict, Mapping
import flask
from flask import jsonify
# import parsing # original
def post(body: Mapping[str, Any]) -> flask.Response:
html = body["html"]
url: str = body["url"]
headers: Dict[str, str] = body.get("headers", {})
#text = parsing.parse_html(html, headers, url) # original
text = "hi" # dummy so it works for you
return cast(flask.Response, jsonify({"text": text}))
test code:
# test_parse.py
import pytest
from app import app as connexion_app
BASE_64_IMAGE = "R0lGODlhAQABAIAAAP///wAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw=="
BYTES_IMAGE = b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00"
INVALID_RESPONSE_JSON_KEYS = {"detail", "type", "status", "title"}
@pytest.fixture
def client():
with connexion_app.test_client() as client:
return client
@pytest.mark.parametrize(
"data",
[BASE_64_IMAGE, BYTES_IMAGE],
)
def test_post_with_image_data_returns_415(data, client):
response = client.post("/path", data=data)
assert set(response.json()) == INVALID_RESPONSE_JSON_KEYS
assert response.status_code == 415
"app" code
# app.py
import connexion
app = connexion.FlaskApp(__name__)
app.add_api(
specification="api.yaml",
strict_validation=True,
validate_responses=True,
)
run
pytest
Additional info:
Old versions: Python 3.10.6 connexion==2.14.0
For old version you have to change code slightly response.json() to response.get_json() and fixture to:
@pytest.fixture
def client():
return connexion_app.app.test_client()
New versions: Python 3.12.1 connexion==3.0.5
The only place I have found that should be fixed/changed is https://github.com/spec-first/connexion/blob/211bdb03f6b20f7dc90fe0037e57cd142c48deb5/connexion/validators/json.py#L54 The main issue is that
body = bytes_body.decode(self._encoding)
should we wrapped in try-except like:
try:
body = bytes_body.decode(self._encoding)
except Exception:
body = None # this or return None
then 500 in one test case will change to 400 so both will return 400 instead 415.
Changing exceptions raised by this method from BadRequestProblem to UnsupportedMediaTypeProblem won't help to bring back previous behaviors as it seems nothing catches UnsupportedMediaTypeProblem in calling code because 500 status code is returned instead 400 or 415.