Unpickling custom exceptions

[dbmikus]

When I raise a custom exception inside DBOS, I get an error unpickling that exception. I'm not sure if this happens broadly with exceptions that have custom __init__ methods.

Here is an example workflow:

from fastapi import APIRouter, HTTPException, status as httpstatus

@DBOS.workflow()
async def my_wf() -> List[AuthorSkillsCheckResult]:
    raise HTTPException(status_code=httpstatus.HTTP_500_INTERNAL_SERVER_ERROR, detail="DIE DIE DIE")

Then I call DBOS.retrieve_workflow like so:

async def get_workflow(workflow_id: str) -> WorkflowResult[Any]:
    """Get the workflow status, and if finished, the result"""
    try:
        handle: WorkflowHandle[Any] = await asyncio.to_thread(
            DBOS.retrieve_workflow, workflow_id
        )
    except dbos_error.DBOSNonExistentWorkflowError as e:
        raise HTTPException(status_code=404, detail="Workflow not found") from e

    ... # rest of my code

And here is the exception I get:

  File "/home/dbmikus/workspace/github.com/gofixpoint/fixpoint/src/server/fixpoint_server/datajobs/research_papers/workflows/utils/wf.py", line 42, in get_workflow
    handle: WorkflowHandle[Any] = await asyncio.to_thread(
                                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linuxbrew/.linuxbrew/Cellar/[email protected]/3.12.3/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linuxbrew/.linuxbrew/Cellar/[email protected]/3.12.3/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dbmikus/.cache/pypoetry/virtualenvs/fixpoint-server-Jyuct7p8-py3.12/lib/python3.12/site-packages/dbos/_dbos.py", line 765, in retrieve_workflow
    stat = dbos.get_workflow_status(workflow_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dbmikus/.cache/pypoetry/virtualenvs/fixpoint-server-Jyuct7p8-py3.12/lib/python3.12/site-packages/dbos/_dbos.py", line 749, in get_workflow_status
    return _get_dbos_instance()._sys_db.call_function_as_step(fn, "DBOS.getStatus")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dbmikus/.cache/pypoetry/virtualenvs/fixpoint-server-Jyuct7p8-py3.12/lib/python3.12/site-packages/dbos/_sys_db.py", line 1852, in call_function_as_step
    result = fn()
             ^^^^
  File "/home/dbmikus/.cache/pypoetry/virtualenvs/fixpoint-server-Jyuct7p8-py3.12/lib/python3.12/site-packages/dbos/_dbos.py", line 747, in fn
    return get_workflow(_get_dbos_instance()._sys_db, workflow_id, True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dbmikus/.cache/pypoetry/virtualenvs/fixpoint-server-Jyuct7p8-py3.12/lib/python3.12/site-packages/dbos/_workflow_commands.py", line 84, in get_workflow
    infos: List[WorkflowStatus] = sys_db.get_workflows(input, get_request)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dbmikus/.cache/pypoetry/virtualenvs/fixpoint-server-Jyuct7p8-py3.12/lib/python3.12/site-packages/dbos/_sys_db.py", line 883, in get_workflows
    info.error = _serialization.deserialize_exception(row[18])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dbmikus/.cache/pypoetry/virtualenvs/fixpoint-server-Jyuct7p8-py3.12/lib/python3.12/site-packages/dbos/_serialization.py", line 54, in deserialize_exception
    upo: Exception = jsonpickle.decode(serialized_data)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dbmikus/.cache/pypoetry/virtualenvs/fixpoint-server-Jyuct7p8-py3.12/lib/python3.12/site-packages/jsonpickle/unpickler.py", line 103, in decode
    return context.restore(data, reset=reset, classes=classes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dbmikus/.cache/pypoetry/virtualenvs/fixpoint-server-Jyuct7p8-py3.12/lib/python3.12/site-packages/jsonpickle/unpickler.py", line 420, in restore
    value = self._restore(obj)
            ^^^^^^^^^^^^^^^^^^
  File "/home/dbmikus/.cache/pypoetry/virtualenvs/fixpoint-server-Jyuct7p8-py3.12/lib/python3.12/site-packages/jsonpickle/unpickler.py", line 402, in _restore
    return restore(obj)
           ^^^^^^^^^^^^
  File "/home/dbmikus/.cache/pypoetry/virtualenvs/fixpoint-server-Jyuct7p8-py3.12/lib/python3.12/site-packages/jsonpickle/unpickler.py", line 555, in _restore_reduce
    stage1 = f(*args)
             ^^^^^^^^
TypeError: HTTPException.__init__() missing 1 required positional argument: 'status_code'

[kraftp]

Yes, unfortunately this is an issue with pickling exceptions in general. For example, see this issue: https://github.com/python/cpython/issues/76877

The solution is to provide custom __reduce__ methods for your custom exceptions to make them picklable. For example, we do that here for exceptions that can be thrown from workflows: https://github.com/dbos-inc/dbos-transact-py/blob/main/dbos/_error.py

[dbmikus]

Thanks for the pointer. Is this documented on https://docs.dbos.dev/ ? I don't recall seeing documentation, and think it would be nice to add!

[kraftp]

Yeah, that's a good idea—was thinking of putting together a "Troubleshooting" page for issues like this.

[larsblumberg]

I'm having the same issues with steps that have exceptions thrown by 3rd libraries which don't implement __reduce__ yet. Quite annoying during workflow resume.

[kraftp]

The underlying issue is unfortunately inherent to pickle. Added support for custom serialization, which may help with this: https://github.com/dbos-inc/dbos-transact-py/pull/486