avalanche
avalanche copied to clipboard
COPE does not work anymore
The COPE strategy is not working on the Split MNIST benchmark on which it was originally tested (it achieves a performance around 20% accuracy instead of 94%).
After some experiments, I figured out that the commit producing the regression is: e149678ca5d467671fc7800c18633f3678d74d8d (July 28, 2021 with title "COPE fix, only train 1 iteration per batch")
This commit introduced two major changes:
-
max_it_cnt
, to avoid training multiple times on the same minibatch (it is fixed to 1) - the usage of
data_incremental_benchmark
to build the benchmark out of Split MNIST. Before, the strategy worked on the usual stream with 5 experiences.
I changed the former from 1 to 10 without significant difference in the final result. The problem may then be related to the second point.
I still did not figure out how to fix this but in light of the upcoming release I think we need to decide what to do. If we are not able to fix it in time, the user should be warned about the bug when using COPE.
I added a warning for COPE in #1055 The bug still needs to be fixed
fixed in #1285