avalanche icon indicating copy to clipboard operation
avalanche copied to clipboard

Still checkpointing problems

Open wistuba opened this issue 1 year ago • 1 comments

I reported earlier in #1269 that the example wasn't working and I just checked the latest version on main to see whether it works and I get

(avalanche) 147ddaa47ac3:examples marwistu$ python checkpointing.py --cuda -1
Using device cpu
Traceback (most recent call last):
  File "/Users/marwistu/avalanche/examples/checkpointing.py", line 96, in <module>
    main_with_checkpointing(parser.parse_args())
  File "/Users/marwistu/avalanche/examples/checkpointing.py", line 61, in main_with_checkpointing
    strategy = Naive(
  File "/Users/marwistu/miniconda/miniconda3/envs/avalanche/lib/python3.9/site-packages/avalanche/training/supervised/strategy_wrappers.py", line 100, in __init__
    super().__init__(
  File "/Users/marwistu/miniconda/miniconda3/envs/avalanche/lib/python3.9/site-packages/avalanche/training/templates/common_templates.py", line 128, in __init__
    super().__init__()  # type: ignore
TypeError: __init__() missing 2 required positional arguments: 'model' and 'optimizer'

Besides that, I still cannot load checkpoints as mentioned in #1270 with the latest Pytorch 1 version as also observed by someone else in that issue. I wasn't able to replicate with a simple example as above since this isn't working at the moment. My code works with Pytorch 1.12 but I would like to upgrade to 1.13 for some other dependencies I am using.

wistuba avatar May 23 '23 15:05 wistuba

Hey @wistuba, sorry for the late reply. I tested on pytorch==2.0.1+cu117 and the example is working on cpu, gpu, and with num_workers>0 (saving and resuming). We did several fixes over the last few months.

AttributeError: Can't pickle local object 'EmptyTransformGroups.init..'

this was fixed by a previous commit.

TypeError: init() missing 2 required positional arguments: 'model' and 'optimizer'

This was a backward-incompatible change in typing-extensions, unrelated to checkpointing. If you install on a fresh environment you shouldn't see this error. Otherwise just pip install typing-extensions==4.4.0.

Let me know if you still have issues. If the problem is with a specific pytorch version, I'm afraid that for the moment we can't really support it. We just don't have the manpower to check incompatibilities between minor versions of pytorch. We can add a warning for that specific version.

AntonioCarta avatar Jul 26 '23 08:07 AntonioCarta