aws-powertools
Bug: RequestValidationError handling behavior should not be affected if we add exception handler for Exception
Expected Behaviour
I have an exception handler for Exception.
app = APIGatewayRestResolver(enable_validation=True)
@app.exception_handler(Exception)
def handle_exception(ex: Exception):
...
When a validation error happens, I want to get the response:
{
"detail": [
{
"loc": ["body", "message"],
"type": "missing",
},
{"loc": ["body", "sendTo"], "type": "value_error"},
],
"statusCode": 422,
},
Current Behaviour
I get the response of my error handler instead of the pydantic errors.
{'message': 'internal server error'}
Code snippet
Copy paste of above snippet:
@app.exception_handler(Exception)
def handle_exception(ex: Exception):
....
Possible Solution
The code responsible is
class ApiGatewayResolver(BaseRouter):
....
def _lookup_exception_handler(self, exp_type: Type) -> Optional[Callable]:
# Use "Method Resolution Order" to allow for matching against a base class
# of an exception
for cls in exp_type.__mro__:
if cls in self._exception_handlers:
return self._exception_handlers[cls]
return None
def _call_exception_handler(self, exp: Exception, route: Route) -> Optional[ResponseBuilder]:
handler = self._lookup_exception_handler(type(exp))
if handler:
try:
return self._response_builder_class(response=handler(exp), serializer=self._serializer, route=route)
except ServiceError as service_error:
exp = service_error
if isinstance(exp, RequestValidationError):
# For security reasons, we hide msg details (don't leak Python, Pydantic or file names)
errors = [{"loc": e["loc"], "type": e["type"]} for e in exp.errors()]
return self._response_builder_class(
response=Response(
status_code=HTTPStatus.UNPROCESSABLE_ENTITY,
content_type=content_types.APPLICATION_JSON,
body={"statusCode": HTTPStatus.UNPROCESSABLE_ENTITY, "detail": errors},
),
serializer=self._serializer,
route=route,
)
My workaround is to have a version of APIGatewayRestResolver that avoids returning error handler for Exception for RequestValidationError.
from aws_lambda_powertools.event_handler import APIGatewayRestResolver as BaseAPIGatewayRestResolver
from aws_lambda_powertools.event_handler.openapi.exceptions import RequestValidationError
...
class APIGatewayRestResolver(BaseAPIGatewayRestResolver):
def _lookup_exception_handler(self, exp_type: typing.Type) -> typing.Optional[typing.Callable]:
if exp_type == RequestValidationError:
return None
return super()._lookup_exception_handler(exp_type)
app = APIGatewayRestResolver(enable_validation=True)
I understand my workaround may be against the behavior of https://github.com/aws-powertools/powertools-lambda-python/issues/3395 .
If I had to suggest a fix, I would suggest moving validation error from _call_exception_handler to its own method, similar to the case of _not_found handler:
def _not_found(self, method: str) -> ResponseBuilder:
"""Called when no matching route was found and includes support for the cors preflight response"""
....
handler = self._lookup_exception_handler(NotFoundError)
if handler:
return self._response_builder_class(response=handler(NotFoundError()), serializer=self._serializer)
return self._response_builder_class(
response=Response(
status_code=HTTPStatus.NOT_FOUND.value,
content_type=content_types.APPLICATION_JSON,
headers=headers,
body={"statusCode": HTTPStatus.NOT_FOUND.value, "message": "Not found"},
),
serializer=self._serializer,
)
Steps to Reproduce
Copy paste of above snippet:
@app.exception_handler(Exception)
def handle_exception(ex: Exception): # pragma: no cover
Then trigger a pydantic validation error.
Powertools for AWS Lambda (Python) version
latest
AWS Lambda function runtime
3.8
Packaging format used
PyPi
Debugging logs
No response
Thanks for opening your first issue here! We'll come back to you as soon as we can. In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link
![boring-cyborg[bot] avatar](https://avatars.githubusercontent.com/in/50386?v=4 "Published by a pub.dev verified publisher"){kind=small}
hey @aitchnyu - thank you for submitting an extensive bug report!! It should work with RequestValidationError exception.
I'll investigate the behaviour today when catching a catch-all Exception like you did, and come back to you.
As per docs, catching RequestValidatioError should work as you want; tested locally too.
from typing import Optional
import requests
from pydantic import BaseModel, Field
from aws_lambda_powertools import Logger, Tracer
from aws_lambda_powertools.event_handler import APIGatewayRestResolver, Response, content_types
from aws_lambda_powertools.event_handler.openapi.exceptions import RequestValidationError
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.utilities.typing import LambdaContext
tracer = Tracer()
logger = Logger()
app = APIGatewayRestResolver(enable_validation=True)
class Todo(BaseModel):
userId: int
id_: Optional[int] = Field(alias="id", default=None)
title: str
completed: bool
@app.exception_handler(RequestValidationError)
def handle_validation_error(ex: RequestValidationError):
logger.error("Request failed validation", path=app.current_event.path, errors=ex.errors())
return Response(
status_code=422,
content_type=content_types.APPLICATION_JSON,
body="Invalid data",
)
@app.post("/todos")
def create_todo(todo: Todo) -> int:
response = requests.post("https://jsonplaceholder.typicode.com/todos", json=todo.dict(by_alias=True))
response.raise_for_status()
return response.json()["id"]
@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_HTTP)
@tracer.capture_lambda_handler
def lambda_handler(event: dict, context: LambdaContext) -> dict:
return app.resolve(event, context)
Can't reproduce with either general Exception and the suggested RequestValidationException -- need more details.
Example below on how to get those exact details what the documentation suggests with RequestValidationError.
from typing import Annotated, Optional
import requests
from aws_lambda_powertools import Logger, Tracer
from aws_lambda_powertools.event_handler import (
APIGatewayRestResolver,
Response,
content_types,
)
from aws_lambda_powertools.event_handler.openapi.exceptions import (
RequestValidationError,
)
from aws_lambda_powertools.event_handler.openapi.params import Path, Query
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools.utilities.parser import Field, BaseModel
tracer = Tracer()
logger = Logger()
app = APIGatewayRestResolver(enable_validation=True)
app.enable_swagger(path="/swagger")
class Todo(BaseModel):
userId: int
id_: Optional[int] = Field(alias="id", default=None)
title: str
completed: bool
@app.exception_handler(RequestValidationError)
def catch_validation(ex: RequestValidationError):
logger.info("Catch all exception handler", error=ex)
err = ex.errors()[0]
location, message, type = err["loc"], err["msg"], err["type"]
return Response(
status_code=400,
content_type=content_types.TEXT_PLAIN,
body=f"Uh oh! Received an exception",
)
@app.get("/todos")
@tracer.capture_method
def get_todos(
completed: Annotated[str | None, Query(min_length=4)] = None
) -> list[Todo]:
url = "https://jsonplaceholder.typicode.com/todos"
if completed is not None:
url = f"{url}/?completed={completed}"
todo = requests.get(url)
todo.raise_for_status()
return todo.json()
@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
@tracer.capture_lambda_handler
def lambda_handler(event: dict, context: LambdaContext) -> dict:
return app.resolve(event, context)
My workaround is having handlers for both RequestValidationError and Exception. If I were to disable the RequestValidationError handler, the Exception would handle it (against my expectations).
app = APIGatewayRestResolver(enable_validation=True)
...
@app.exception_handler(Exception)
def handle_exception(ex: Exception): # pragma: no cover
...
return Response(
status_code=HTTPStatus.INTERNAL_SERVER_ERROR,
content_type=content_types.APPLICATION_JSON,
body=json.dumps({"message": "internal server error"}),
)
@app.exception_handler(RequestValidationError)
def handle_request_validation_error(ex: RequestValidationError):
# Copy of ApiGatewayResolver _call_exception_handler
errors = [{"loc": e["loc"], "type": e["type"]} for e in ex.errors()]
return Response(
status_code=HTTPStatus.UNPROCESSABLE_ENTITY,
content_type=content_types.APPLICATION_JSON,
body={"statusCode": HTTPStatus.UNPROCESSABLE_ENTITY, "detail": errors},
)
Hi @heitorlessa and @aitchnyu, after investigating further I discovered that the issue goes beyond fixing the RequestValidationError or not. Our exception_handler function captures a list of exceptions passed to it, however, it is important to note that most exceptions in Python inherit from Exception (some from BaseException). So if we set exception_handler to catch Exception, it will also catch exceptions like ValueError, KeyError, TypeError, RequestValidationError, and others.
Ideally, it's advisable to catch more specific exceptions, especially if you want to generate custom messages for specific error types like RequestValidationError. I'm not sure if there's a straightforward solution at the moment, I'll need to investigate further. One potential action arising from this issue is to update our documentation to reflect this behavior, where catching Exception acts as a catch-all mechanism for all exceptions.
code
from aws_lambda_powertools import Logger, Tracer
from aws_lambda_powertools.event_handler import APIGatewayRestResolver
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools.event_handler.api_gateway import Response
tracer = Tracer()
logger = Logger()
app = APIGatewayRestResolver()
@app.get("/hello")
def get_todos():
raise ValueError
@app.exception_handler(Exception)
def handle_invalid_limit_qs(ex: Exception): # receives exception raised
return Response(
status_code=400,
body="Catching Exception, but raising ValueError",
)
@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
def lambda_handler(event: dict, context: LambdaContext) -> dict:
return app.resolve(event, context)
execution
sam local invoke --event events/event.json --skip-pull-image
Invoking app.lambda_handler (python3.11)
Requested to skip pulling images ...
Mounting /home/leandro/DEVEL-PYTHON/tmp/exception-different-router/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container
START RequestId: e408630c-0660-4d24-a3c5-d6e590f6d959 Version: $LATEST
START RequestId: e408630c-0660-4d24-a3c5-d6e590f6d959 Version: $LATEST
END RequestId: e408630c-0660-4d24-a3c5-d6e590f6d959
REPORT RequestId: e408630c-0660-4d24-a3c5-d6e590f6d959 Init Duration: 0.11 ms Duration: 407.67 ms Billed Duration: 408 ms Memory Size: 128 MB Max Memory Used: 128 MB
{"statusCode": 400, "body": "Catching Exception, but raising ValueError ", "isBase64Encoded": false, "multiValueHeaders": {}}
I'm adding labels such as “need help” and “revisit” to keep this issue on our radar.
Thank you
Hey @heitorlessa and @aitchnyu!
After a long investigation, we have concluded that this is a fundamental aspect of how Python handles exceptions. In Python, most exceptions inherit from the base class Exception, which means that catching Exception will capture a wide range of exception types, including ValueError, KeyError, TypeError, and RequestValidationError. This behavior is an integral part of the Python language design. Modifying how Powertools for AWS Lambda handles and catches exceptions is not a viable solution at this time, and even if we attempted to change it, we have no idea of the potential side effects.
For cases like this, the recommended approach is to catch more specific exceptions, to better handle and respond to different error types.
I'm closing this issue and please reopen if you find a solution or have an idea how to address this.
Thanks
⚠️COMMENT VISIBILITY WARNING⚠️
This issue is now closed. Please be mindful that future comments are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}