PyCIL
PyCIL copied to clipboard
bug in eval_task()
Hello, I've found some bugs about your code computing the accuracy after each task.
After training on the new task, the learner is expected to evaluate on all seen data, and in your code it is done by calling self.eval_task()
: in base.py
line 84:
def eval_task(self, save_conf=False):
y_pred, y_true = self._eval_cnn(self.test_loader)
cnn_accy = self._evaluate(y_pred, y_true)
...
When taking a close look at the _evaluate()
function:
def _evaluate(self, y_pred, y_true):
ret = {}
grouped = accuracy(y_pred.T[0], y_true, self._known_classes)
...
where the function accuracy()
is defined as:
def accuracy(y_pred, y_true, nb_old, increment=10):
assert len(y_pred) == len(y_true), "Data length error."
all_acc = {}
all_acc["total"] = np.around(
(y_pred == y_true).sum() * 100 / len(y_true), decimals=2
)
# Grouped accuracy
for class_id in range(0, np.max(y_true), increment):
Here increment
is used to pick indexs of those old tasks, but in your code you don't pass the param "increment" to the accuracy()
function, which means even I customize the increment in config file, the code fails to evaluate on my pattern. I think you should pass self.args["increment"]
when calling the accuracy()
function.