[Question] Rerun validation without training when I have a consolidated model

[demcbs]

Hi Michael, I had another question.

I would like to check the validation performance of a pretrained model (well, 5, one per fold) which comes only with the consolidated folder. I also have the data and splits it was trained on and preproccesing and unpacking went fine.

I've tried nndet_eval but I get raise ValueError(f"Found wrong number of training dirs {candidates} in {model_dir}"), as the model folder only contains 'consolidated' I suppose.

Is there a way to rerun just the validation in this situation?

Thanks!

[mibaumgartner]

Hi, yes, it should be possible to rerun the evaluation with the predictions and preprocessed data (especially the prepared labels). The error indicates that nndet can not find the model you are trying to evaluate though. To evaluate the consolidated folder you need to pass -1 for the fold, maybe that was the problem?

It is raised here: https://github.com/MIC-DKFZ/nnDetection/blob/c45a49dc7044cd061f2a0e9efc0b0e331485f3d1/nndet/io/paths.py#L228-L245

[demcbs]

I see, that does bring me one step further.

Now I get: FileNotFoundError: [Errno 2] No such file or directory: 'path/consolidated/val_predictions'. The 'consolidated' folder I got only contains the model ckpts, the config.yaml and the plan_inference.pkl.

So I guess a part of the data prep needs to happen (as part of the training and consolidation?).

[mibaumgartner]

If the model was consolidated correctly there should be an additional folder val_predictions which are the predictions from each model on its respective fold. With the checkpoint, config and plan it is only possible to predict and evaluate an external test set. It might be possible to copy each model with the config and plan into the different folds and rerun the validation prediction and consolidation to obtain the predictions again.

[demcbs]

I'll give that a shot, thanks!

[demcbs]

So far, that seems to work, that is, I can run nndet_sweep per fold.

Just to check, can I safely use the plan_inference.pkl and config.yml as plans.pkl and config.yml per fold, respectively, or are they not necessarily the same?

If not, are the plans.pkl and config.yml that are created when starting training the right ones?

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.