rocketry icon indicating copy to clipboard operation
rocketry copied to clipboard
verified publisher icon Miksus

BUG crashed in pickling

Open tekumara opened this issue 3 years ago • 3 comments

Describe the bug

On the second run of my task, the following log line triggers continuously in a loop forever:

CRITICAL:rocketry.task:Task 'daily_notification' crashed in pickling. Cannot pickle: {'__dict__': {'session': <rocketry.session.Session object at 0x2f6d4f9a0>, 'permanent': False, 'fmt_log_message': "Task '{task}' status: '{action}'", 'daemon': None, 'batches': [], 'name': 'daily_notification', 'description': None, 'logger_name': 'rocketry.task', 'execution': None, 'priority': 0, 'disabled': False, 'force_run': False, 'force_termination': False, 'status': 'run', 'timeout': None, 'parameters': Parameters(), 'start_cond': None, 'end_cond': None, 'multilaunch': None, 'on_startup': False, 'on_shutdown': False, 'func_run_id': None, 'func': <function daily_notification at 0x1038cd4c0>, 'path': None, 'func_name': 'main', 'cache': False, 'sys_paths': []}}

The log line is the same each time, except for the Session object reference.

On CTRL+C the trace back contains:

INFO:rocketry.scheduler:Interupted. Shutting down...
Traceback (most recent call last):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/schedule.py", line 128, in serve
    await self.run_cycle()
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/schedule.py", line 182, in run_cycle
    await self.run_task(task)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/schedule.py", line 211, in run_task
    await task.start_async(log_queue=self._log_queue)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 514, in start_async
    self.run_as_process(params=params, direct_params=direct_params, task_run=task_run, **kwargs)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 735, in run_as_process
    process.start()
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1324, in __getstate__
    unpicklable = {key: val for key, val in state.items() if not is_pickleable(val)}
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1324, in <dictcomp>
    unpicklable = {key: val for key, val in state.items() if not is_pickleable(val)}
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1324, in __getstate__
    unpicklable = {key: val for key, val in state.items() if not is_pickleable(val)}
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1324, in <dictcomp>
    unpicklable = {key: val for key, val in state.items() if not is_pickleable(val)}
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1324, in __getstate__
    unpicklable = {key: val for key, val in state.items() if not is_pickleable(val)}
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1324, in <dictcomp>
    unpicklable = {key: val for key, val in state.items() if not is_pickleable(val)}
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1318, in __getstate__
    if not is_pickleable(state):
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/utils/pickle.py", line 6, in is_pickleable
    pickle.dumps(obj)
  File "/Users/tekumara/code/dt-web-glimpse/.venv/lib/python3.9/site-packages/rocketry/core/task.py", line 1326, in __getstate__
    self.logger.critical(f"Task '{self.name}' crashed in pickling. Cannot pickle: {unpicklable}", extra={"action": "fail", "task_name": self.name})
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1835, in critical
    self.log(CRITICAL, msg, *args, **kwargs)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1844, in log
    self.logger.log(level, msg, *args, **kwargs)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1512, in log
    self._log(level, msg, args, **kwargs)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 952, in handle
    self.emit(record)
  File "/Users/tekumara/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1086, in emit
    stream.write(msg + self.terminator)
KeyboardInterrupt
INFO:rocketry.scheduler:Shutdown completed. Good bye.

My task:

@app.task(hourly)
async def daily_notification() -> None:
    file = "summary.png"
    # we are using the sync clients so run them on a thread
    await to_thread.run_sync(
        glance.browser.screenshot,
        file
    )
    await to_thread.run_sync(glance.slack.upload_screenshot, file)

Desktop (please complete the following information):

  • OS: macos
  • Python version 3.9

Additional context

rocketry 2.5.1

tekumara avatar Dec 11 '22 11:12 tekumara

As a workaround, switching the task execution to thread or async (rather than default of process) avoids this problem, presumably because pickling doesn't need to happen.

tekumara avatar Dec 11 '22 11:12 tekumara

Thanks for reporting this! These pickling issues are often hard to find as they are often OS-dependent or come up in special situations.

I think an unpicklable attribute from Rocketry's session leaks to the process for some reason.

If interested, the task sort of prunes the session:

  • In the task pickling: https://github.com/Miksus/rocketry/blob/master/rocketry/core/task.py#L1316
  • Pruning the session: https://github.com/Miksus/rocketry/blob/master/rocketry/session.py#L541-L547

If I remember correctly, the process task needs some config options from the session thus most of the session configs are pickled. I tried to make it so that it would throw away only known unpicklable attributes, but I am not sure if it should work the other way around (including only the ones we know we can pickle).

Another could be the function but that function looks pretty standard.

How are your app configurations, and do you perhaps set some attributes to the session?

Miksus avatar Dec 11 '22 12:12 Miksus

Thanks for the explanation!

This is the only config I have:

app = Rocketry(
    # see https://rocketry.readthedocs.io/en/stable/handbooks/config.html
    config={
        "task_execution": "process",
    }
)

Interestingly, if I remove that config and just use app = Rocketry() it works even though I thought that process was the default.

tekumara avatar Dec 11 '22 21:12 tekumara