INFO: Resuming from iteration for provided data will fetch data until required iteration ...

[H4dr1en]

❓ Questions/Help/Support

I observed in the training logs the message that I don't understand, could you please clarify what happens here and why?

INFO: Resuming from iteration for provided data will fetch data until required iteration ...

This happens for all validation engines I have, that I create as follows:

        for valid_dataset_name, valid_engine in valid_engines.items():
            valid_loader = valid_loaders[valid_dataset_name]
            train_engine.add_event_handler(Events.EPOCH_COMPLETED, partial(valid_engine.run, data=valid_loader))

Note: I use DeterministicEngine for all engines (training and validation)

[vfdev-5]

The message means that deterministic engine is trying to resume the run from some non-zero iteration. For deterministic engines we have to rewind dataloader up to the resuming iteration otherwise randomness state wont be fully respected (probably here there can be more context: https://pytorch.org/ignite/engine.html#dataflow-synchronization).

Given the code you provide, I would say this is more like a bug. Probably, valid_engine was stopped at some point without getting the full 1 epoch and then it was called to run again...

[H4dr1en]

Given the code you provide, I would say this is more like a bug.

Do you mean a bug in ignite or in my code?

Probably, valid_engine was stopped at some point without getting the full 1 epoch and then it was called to run again...

I am not stopping any of the valid_engines, they all run for a single full epoch of validation after each training epoch

[vfdev-5]

Do you mean a bug in ignite or in my code?

Difficult to say like that. Is it possible that you could provide more code to repro the issue ?