anomalib icon indicating copy to clipboard operation
anomalib copied to clipboard

Error for multiple gpu training

Open udasinnayan opened this issue 2 years ago • 2 comments

In the config file trainer part, if I set auto_select_gpus: true and devices: 2, I got an error like AttributeError: Can't pickle local object 'FeatureExtractor.get_features.<locals>.hook', it comes from this line trainer.fit(model=model, datamodule=datamodule) in anomalib/tools/train.py script but for single GPU it works without an error. Any comments about this error / is it possible to use multiple GPUs for training and testing?

udasinnayan avatar Jul 12 '22 12:07 udasinnayan

Hi @udasinnayan that's a known issue. This might probably be solved when me move towards torchfx but the only option is to use single GPU for now.

ashwinvaidya17 avatar Jul 12 '22 15:07 ashwinvaidya17

@ashwinvaidya17, I'm wondering if this is the case even when we use timm's feature extractor

samet-akcay avatar Jul 13 '22 05:07 samet-akcay

I tested this on the current main branch and it still does not work with timm's feature extractor. I am closing as it will require a significant change.

ashwinvaidya17 avatar Sep 14 '22 14:09 ashwinvaidya17