BackgroundTasks do not run when request failed

[NewLanded]

Hello everybody, I have a question about background task, there is an example

This is running fine.

@app.post("/")
def user(background_tasks: BackgroundTasks):
    background_tasks.add_task(func_example)

But when I raise an exception, the background task will not be running.

@app.post("/")
def user(background_tasks: BackgroundTasks):
    background_tasks.add_task(func_example)
    raise HTTPException(status_code=500, detail="example error")

I want to know this is a bug or a feature? Thanks for any answer.

[krishnardt]

Background tasks will run only after the response(after the response is successful).

Once the request is received, the task related to the request will be added to the background task. Once the response is successful, the task added to Background will be executed. if the repsonse is a failure due to any issue(whether it can be input validation issue, internal server error, raising exceptions), task will not be executed.

to keep it simple, that is a feature to avoid running the task, if api fails in any case.

[Kludex]

It is a feature.

| You can define background tasks to be run after returning a response. "Returning" here is the literal return in the endpoint function. If you never reach it, it will not run the task.

Reference: https://fastapi.tiangolo.com/tutorial/background-tasks/#background-tasks

[Kludex]

I think we should be more explicit in the docs about this. :disappointed:

[iamthen0ise]

I think it is better to think about BackgroundTask as a defer function, which run just after the successful call of parent one.

It is not even close to any existing Task Queue implementation. Think of Celery, Rq, or even self-made queues if you need to have all benefits of it — retries, backoff, pub/sub, etc

Otherwise, think about BackgroundTask as a some way to run a successor function just make the things done, right after the request is succeed. E.g. update cache, alter user session attributes, and so on.

[amoghmishra-sl]

Is there a workaround for this? I'd want to generate the error logs as a background task when there is a failure. Given the current feature, they aren't getting generated.

[NewLanded]

Background tasks will run only after the response(after the response is successful).

Once the request is received, the task related to the request will be added to the background task. Once the response is successful, the task added to Background will be executed. if the repsonse is a failure due to any issue(whether it can be input validation issue, internal server error, raising exceptions), task will not be executed.

to keep it simple, that is a feature to avoid running the task, if api fails in any case.

Thanks a lot. The answer was very clear and cleared up my doubts

[NewLanded]

I tried the code and it works for me. background worked before 'return JSONResponse', This is my script

import logging import time import traceback from logging import handlers

from fastapi import FastAPI, Request from fastapi.responses import JSONResponse from starlette.background import BackgroundTask

logger = logging.getLogger('./test.log') logger.setLevel(logging.INFO) formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(name)s - %(filename)s - %(funcName)s - %(lineno)s - %(message)s')

rf = handlers.RotatingFileHandler('./test.log', encoding='UTF-8', maxBytes=1024, backupCount=0) rf.setLevel(logging.INFO) rf.setFormatter(formatter)

logger.addHandler(rf)

app = FastAPI()

@app.middleware("http") async def middleware(request: Request, call_next): start_time = time.time() try: response = await call_next(request) process_time = time.time() - start_time response.headers["X-Process-Time"] = str(process_time) raise ValueError("test") except Exception as exc: background = BackgroundTask(write_logs, message=f"{traceback.format_exc()}") return JSONResponse(status_code=500, content={"reason": traceback.format_exc()}, background=background)

def write_logs(message): logger.info(message)

@app.get("/") async def index(): return "test"

if name == "main": import uvicorn

uvicorn.run(app, host="0.0.0.0", port=6677)

@.***

From: amoghmishra-sl Date: 2022-08-09 01:46 To: tiangolo/fastapi CC: NewLanded; Author Subject: Re: [tiangolo/fastapi] BackgroundTasks do not run when request failed (#2604) Is there a workaround this? I'd want to generate the error logs as a background task when there is a failure. Given the current feature, they aren't getting generated. I am trying to do something like this: import time import traceback from fastapi import FastAPI, Request from fastapi.responses import JSONResponse from starlette.background import BackgroundTask app = FastAPI() @app.middleware("http") async def middleware(request: Request, call_next): start_time = time.time() try: response = await call_next(request) process_time = time.time() - start_time response.headers["X-Process-Time"] = str(process_time) except Exception as exc: background = BackgroundTask(write_log, message=f"{traceback.format_exc()}") return JSONResponse(status_code=500, content={"reason": traceback.format_exc()}, background = background)

where write_log is something like: def write_logs(message): logger.log(message)

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[amoghmishra-sl]

@NewLanded yeah, this would work as it has the "return" statement. If you add a background task in a scenario like this then it won't work.

@app.post("/send-notification/{email}")
async def send_notification(email: str, background_tasks: BackgroundTasks):
    if "hello" in email:
        background_tasks.add_task(write_notification, message="helloworld")
        raise HTTPException(status_code=500, detail="example error")

    background_tasks.add_task(write_notification, message="hello world.")
    return {"message": "Notification sent in the background"}

[JarroVGIT]

@amoghmishra-sl actually this works because the Backgroundtask is always called after your endpoint returned (even if it returned a HTTPException. Note that this is running as a middleware and is completely separate from calling Backgroundtask in a FastAPI endpoint.

[amoghmishra-sl]

@JarroVGIT The latest example is not of a middleware and BackgroundTasks of FastAPI won't log the error because it wasn't "returned" but raised. As mentioned by @Kludex above,

It is a feature.

| You can define background tasks to be run after returning a response. "Returning" here is the literal return in the endpoint function. If you never reach it, it will not run the task.

Reference: https://fastapi.tiangolo.com/tutorial/background-tasks/#background-tasks

[JarroVGIT]

Oh I thought you were responding to NewLanded's comment right above yours, where he did show a middleware implementation:

@app.middleware("http")
async def middleware(request: Request, call_next):
    start_time = time.time()
    try:
        response = await call_next(request)
        process_time = time.time() - start_time
        response.headers["X-Process-Time"] = str(process_time)
        raise ValueError("test")
    except Exception as exc:
        background = BackgroundTask(write_logs, message=f"{traceback.format_exc()}")
        return JSONResponse(status_code=500, content={"reason": traceback.format_exc()}, background=background)

[amoghmishra-sl]

Np, this worked for me too because the code flow reached the "return" statement. Hence, the background task logged the error.

[JarroVGIT]

Np, this worked for me too because the code flow reached the "return" statement. Hence, the background task logged the error.

I don't think you fully understand why this is working yet, but I am glad it worked for you.

(To be clear: Kludex was referring to the literal return in the FastAPI endpoint. Whatever that returns (either the return of the endpoint, or a HTTPException object, it really does not matter), it will be intercepted by the middleware of @NewLanded. The middleware will raise its own exception, which will be caught and the handler (except:) will run the background task. Note that the middleware here is a ASGI application in itself, just like Starlette and FastAPI is. So, the background task implementation of @NewLanded is different than the context the comment of Kludex was made.)