AttributeError: Can't pickle local object 'EncoderDecoderModelAdaptersMixin._init_adapter_modules.<locals>.<lambda>'

[ZeguanXiao]

Environment info

• adapter-transformers version: 3.1.0
• Platform: Ubuntu 18.04 (Linux-5.4.0-87-generic-x86_64-with-glibc2.27)
• Python version: Python 3.9.13
• PyTorch version (GPU?): 1.13.1 (GPU)
• Tensorflow version (GPU?): False
• Using GPU in script?: True
• Using distributed or parallel set-up in script?: Yes

Information

Model I am using (Bert, XLNet ...): EncoderDecoderModel

Language I am using the model on (English, Chinese ...): English

Adapter setup I am using (if any): AdapterConfig

The problem arises when using:

• [ ] the official example scripts: (give details below)
• [x] my own modified scripts: (give details below)

The tasks I am working on is:

• [ ] an official GLUE/SQUaD task: (give the name)
• [x] my own task or dataset: (give details below)

EncoderDecoderModel can't work in DDP due to being unable to pickle any `self.decoder.base_model.* methods in EncoderDecoderModelAdaptersMixin._init_adapter_modules().

 File "/home/xzg/anaconda3/lib/python3.9/site-packages/hydra/main.py", line 90, in decorated_main
    _run_hydra(
  File "/home/xzg/anaconda3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 389, in _run_hydra
    _run_app(
  File "/home/xzg/anaconda3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 452, in _run_app
    run_and_report(
  File "/home/xzg/anaconda3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 216, in run_and_report
    raise ex
  File "/home/xzg/anaconda3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 213, in run_and_report
    return func()
  File "/home/xzg/anaconda3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 453, in <lambda>
    lambda: hydra.run(
  File "/home/xzg/anaconda3/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/home/xzg/anaconda3/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/home/xzg/anaconda3/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/xzg/triangular-transfer/src/train.py", line 26, in main
    metric_dict, _ = train(cfg)
  File "/home/xzg/triangular-transfer/src/utils/utils.py", line 42, in wrap
    raise ex
  File "/home/xzg/triangular-transfer/src/utils/utils.py", line 39, in wrap
    metric_dict, object_dict = task_func(cfg=cfg)
  File "/home/xzg/triangular-transfer/src/tasks/train_task.py", line 68, in train
    trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
  File "/home/xzg/anaconda3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
    call._call_and_handle_interrupt(
  File "/home/xzg/anaconda3/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/home/xzg/anaconda3/lib/python3.9/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py", line 113, in launch
    mp.start_processes(
  File "/home/xzg/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 189, in start_processes
    process.start()
  File "/home/xzg/anaconda3/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/home/xzg/anaconda3/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/xzg/anaconda3/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/xzg/anaconda3/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/xzg/anaconda3/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/xzg/anaconda3/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'EncoderDecoderModelAdaptersMixin._init_adapter_modules.<locals>.<lambda>'