truss icon indicating copy to clipboard operation
truss copied to clipboard

Allow specification of HTTP status code in predict() return.

Open RF5 opened this issue 1 year ago • 0 comments

Is your feature request related to a problem? Please describe.

If a prediction call fails, you cannot indicate to a caller how or why it failed. The status code is always either HTTP200 OK, or HTTP500 (if any exception is raised, even if you raise a custom Starlette HTTPError exception).

Describe the solution you'd like

The truss async def predict(self, request: Dict) function should allow the user to return a custom HTTP return code. This can be done (I think?) simply by adding the following code:

if isinstance(response, Response): 
    # user-provided predict function returned already packaged response, directly return it. 
    return response

To line 156 of the common truss server here: https://github.com/basetenlabs/truss/blob/main/truss/templates/server/common/truss_server.py#L156

This way, a user-specified function can return a prepackaged Starlette Response object, with custom HTTP return code and header configuration. In addition to allowing the user to now return better error or accept status_codes (e.g. an HTTP 202 ACCEPTED response) , it also allows more advanced users to specify a custom way to packaging their response and its associated headers.

Describe alternatives you've considered

  • Tried to return internal status code inside the regular response dict of predict() function: this is too hacky and requires clients to check an internal status_code field of an HTTP200 OK response to double check if it is a satisfactory response.
  • Tried to raise a Starlette HTTPError or Response objects in the object caller, or to return them instead of raise them. These all result in an HTTP500 internal server error being thrown, not correctly returning the HTTPError or Response object raised/returned.

Additional context The goal here is to allow more advanced users to specify an HTTP status code in the response, and possibly insert custom headers or craft a non-trivial Starlette response for their custom use case.

RF5 avatar Jun 13 '24 08:06 RF5