loguru icon indicating copy to clipboard operation
loguru copied to clipboard

Pickle error on using loguru with ray

Open neelabalan opened this issue 2 years ago • 0 comments

Hi. I'm facing this error on running loguru with ray remote. I think I'm missing something here. I went through the documentation and I couldn't find anything helpful to solve my specific issue. I noticed from issue #386 that @Delgan has mentioned "The logger is picklable as long as the added handlers are picklable too". Is it because of this? Appreicate the help. Thanks for the wonderful library. Loguru has made logging very easy.

Code

from loguru import logger
import ray

logger.add(f'logs/{__name__}', filter=__name__, rotation='1 week')
@ray.remote
class StartUp:
    def start(self):
        for i in range(10000):
            logger.info(i)

def run():
    logger.info("started!")
    agents = [StartUp.remote() for i in range(4)]
    result = []
    for count , agent in enumerate(agents):
        logger.info(count)
        result.append(agent.start.remote())
    done_id, _ = ray.wait(result)
    logger.info("complete!")


if __name__ == "__main__":
    run()

Dependencies

[tool.poetry.dependencies]
python = "3.8.13"
ray = {extras = ["default"], version = "1.13.0"}
loguru = "0.6.0"

Error

Traceback (most recent call last):
  File "test.py", line 23, in <module>
    run()
  File "test.py", line 13, in run
    agents = [StartUp.remote() for i in range(4)]
  File "test.py", line 13, in <listcomp>
    agents = [StartUp.remote() for i in range(4)]
  File "/home/blue/.cache/pypoetry/virtualenvs/loguru-ray-test-l4hNMfu6-py3.8/lib/python3.8/site-packages/ray/actor.py", line 529, in remote
    return self._remote(args=args, kwargs=kwargs, **self._default_options)
  File "/home/blue/.cache/pypoetry/virtualenvs/loguru-ray-test-l4hNMfu6-py3.8/lib/python3.8/site-packages/ray/util/tracing/tracing_helper.py", line 387, in _invocation_actor_class_remote_span
    return method(self, args, kwargs, *_args, **_kwargs)
  File "/home/blue/.cache/pypoetry/virtualenvs/loguru-ray-test-l4hNMfu6-py3.8/lib/python3.8/site-packages/ray/actor.py", line 844, in _remote
    worker.function_actor_manager.export_actor_class(
  File "/home/blue/.cache/pypoetry/virtualenvs/loguru-ray-test-l4hNMfu6-py3.8/lib/python3.8/site-packages/ray/_private/function_manager.py", line 461, in export_actor_class
    serialized_actor_class = pickle.dumps(Class)
  File "/home/blue/.cache/pypoetry/virtualenvs/loguru-ray-test-l4hNMfu6-py3.8/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/home/blue/.cache/pypoetry/virtualenvs/loguru-ray-test-l4hNMfu6-py3.8/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 620, in dump
    return Pickler.dump(self, obj)
  File "/home/blue/.cache/pypoetry/virtualenvs/loguru-ray-test-l4hNMfu6-py3.8/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 320, in _file_reduce
    raise pickle.PicklingError(
_pickle.PicklingError: Cannot pickle files that are not opened for reading: a

neelabalan avatar Sep 15 '22 05:09 neelabalan