ignite

ignite copied to clipboard

🚀 Feature

To add a 'no-improvement' handler. It's similar to early stopping, except that early stopping calls the training to terminate, 'no-improvement' handler would've calls a function that user pass in. User could pass in different functions that they want to execute if they find no improvements is made. They may want to try various things before terminate the training.

@arisliang Thank you for this FR !

Could we imagine that the actual EarlyStopping is just a specific case of the more general NoImprovmentHandler you suggested? I mean NoImprovmentHandler + Stop Policy Function = EarlyStopping? And maybe it's worth considering other trending policies than no-improvment like too-fast-improvment or whatever. What do you think ?

@sdesrozis Thanks for the response.

Yes exactly. I'd imagine EarlyStopping would be a specific case of the more general NoImprovmentHandler by passing in the Stop Policy Function. So NoImprovement determines a scenario where no improvement is made, and decouples the action to be taken, be it stop training, logging, or other things.

I haven't thought about other trending policies. It's certainly possible as another enhancement to make things even more general.

Not sure which is more suitable, no-improvement may even be thought of as an scenario-based event, and actions like stop training can then register as an event handler when no-improvement happens.

@arisliang I will try to write a piece of code to illustrate my thoughts.

Ok so the actual EarlyStopping is the following.

class EarlyStopping:
    
    def __init__(self, patience, score_function, trainer):
        self.score_function = score_function
        self.patience = patience
        self.trainer = trainer
        self.counter = 0
        self.best_score = float('-inf')
        
    def __call__(self, engine):
        score = self.score_function(engine)

        if score <= self.best_score:
            self.best_score = score
            self.counter += 1
            print("EarlyStopping: %i / %i" % (self.counter, self.patience))
            if self.counter >= self.patience:
                print("EarlyStopping: Stop training")
                self.trainer.terminate()
        else:
            self.best_score = score
            self.counter = 0

We could think about a more flexible implementation based on a score_function (as today) associated to a stop_function (i.e. what to do whether no improvment) and a pass_function (i.e. what to do whether improvment).

class TriggerHandlerState:

    def __init__(self):
        self.best_score = float('-inf')
        self.counter = 0

class TriggerHandler:

    def __init__(self, score_function, pass_function, stop_function):
        self.score_function = score_function
        self.pass_function= pass_function
        self.stop_function = stop_function
        self.state = TriggerHandlerState()

    def __call__(self, engine):

        score = self.score_function(engine, self.state)

        if self._update_and_check_state(score):
            self.stop_function(self.state)
        else:
            self.pass_function(self.state)

    def _update_and_check_state(self, score):
        if score <= self.state.best_score:
            self.state.best_score = score
            self.state.counter += 1
            return True
        else:
            return False

What do you think ?

Looks great for both current need and flexible. What is stuck_function? Is it just typo of pass_function?

Looks great for both current need and flexible. What is stuck_function? Is it just typo of pass_function?

Yeah, a typo. Fixed. Let's see whether this design works although the naming, concepts and scope need to be carefully considered.

Hi @sdesrozis and @arisliang, I am interested in taking this up. From what I understand, we should create a Base class called something like NoImprovementHandler and then use it to create the EarlyStopping because it is a special case of that.

I was wondering, should we also keep the decision function of the Base class fixed as if self.counter >= self.patience or are there some use-cases where allowing the user to change this would also help?