How do I provide the continual learning setting?

[prateeky2806]

Hi, thank you for the amazing paper and repository. I am trying to run some experiments on CIFAR100 in task incremental setting with 10 different 10-way classification tasks. Can you please tell me how to specify the continual learning setting for different methods? For example, I see der is compatible with incremental and task-incremental settings so how do I specficy which setting I want to run experiments for? If you can list a few example command it would be very helpful as I don't se an argparse argument for this.

Thanks, Prateek

[mbosc]

Hello there Prateek. In our library, whenever you define a dataset (you can choose one among the available or easily add another), you specify whether it is a Domain-IL (single-head, all classes available at each task), a GCL dataset (see our original paper for the detail) or a Class/Task-IL dataset (different classes in different tasks).

We do not make a proper distinction between Class-IL and Task-IL experiments: if the dataset you use (e.g. CIFAR100) is in this latter category, by running an experiment your model will be tested both for single head accuracy (Class-IL) and for masked single head accuracy (the same as above, but the model can only make a prediction within the set of classes belonging to the target's task). We refer to this latter measure as Task-IL accuracy.

Technically, benchmarking for Task-IL would more properly require that split heads are given at training time. However, we deliberately chose not to do that because this is more efficient for the sake of experiments and we care much more about Class-IL anyway, as Task-IL is a simplified and unrealistic setting.

So - to wrap up - as is the library lets you run Class-IL and Task-IL experiments at once for CIFAR-100. If you're satisfied with our approximation, you can simply take the Task-IL value that you get after training. However, if you really want to work in Task-IL, you should modify the observe method of DER and mask out past and future heads prior to computing the cross-entropy.