Feature: (AutoML) Refactor Trainer for metrics to possibly be measured before training.

[vincent-antaki]

Right now, we only measure metrics after each epoch. It could be useful to measure metrics before the first epoch as to gain better insight on the training process.

Considerations:

• not every pipeline are able to transform before performing a fit. We therefore need to be able to tell the responsible instance (AutoML, Trainer or MetricCallback) whether to perform it or not
• we might want the ability to perform this for some metrics and not for others.
• the TrialMetricPlottingObserver should have access to this information, as to produce the right plot.
• Running the whole callback list before the first epoch might not be desirable/possible; we might need to distinguish between MetricCallback and other callbacks.

[guillaume-chevalier]

@vincent-antaki Here is my suggestion:

• Passing the Trainer to the AutoML constructor will allow to remove some arguments to the constructor of AutoML, as it has too many.
• The new argument to know wheter or not to do this on the first loop is a trainer argument I believe. It seems it is the trainer's job to do this or not.
• Doing so will also prepare for coding Hyperband later on, as coding Hyperband will require coding a new special Hyperband trainer (two birds w/ one stone).

Overall, allowing better injection and customization of the trainer to the AutoML loop will give the expected flexibility.

[guillaume-chevalier]

FYI @Eric2Hamel

[guillaume-chevalier]

See also: https://github.com/Neuraxio/Neuraxle/issues/302

[vincent-antaki]

@guillaume-chevalier I don't mind dependency inversion of Trainer/AutoML. However, your solution does not address point 2 and 3 of the consideration list. I'll implement it anyway because it is sufficient for what I need right now - and rather simple. But keep in mind point 2 and 3 are eventual problems to fix.

[Eric2Hamel]

What is the reason to calculate the metric before the first epoch?

[vincent-antaki]

Prettier training curves plot mainly :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in the next 180 days. Thank you for your contributions.