avalanche icon indicating copy to clipboard operation
avalanche copied to clipboard

COPE does not work anymore

Open AndreaCossu opened this issue 2 years ago • 1 comments

The COPE strategy is not working on the Split MNIST benchmark on which it was originally tested (it achieves a performance around 20% accuracy instead of 94%).
After some experiments, I figured out that the commit producing the regression is: e149678ca5d467671fc7800c18633f3678d74d8d (July 28, 2021 with title "COPE fix, only train 1 iteration per batch")

This commit introduced two major changes:

  1. max_it_cnt, to avoid training multiple times on the same minibatch (it is fixed to 1)
  2. the usage of data_incremental_benchmark to build the benchmark out of Split MNIST. Before, the strategy worked on the usual stream with 5 experiences.

I changed the former from 1 to 10 without significant difference in the final result. The problem may then be related to the second point.

I still did not figure out how to fix this but in light of the upcoming release I think we need to decide what to do. If we are not able to fix it in time, the user should be warned about the bug when using COPE.

AndreaCossu avatar Jun 07 '22 17:06 AndreaCossu

I added a warning for COPE in #1055 The bug still needs to be fixed

AndreaCossu avatar Jun 09 '22 07:06 AndreaCossu

fixed in #1285

AlbinSou avatar Feb 25 '24 15:02 AlbinSou