Delay compilation for as long as possible

[cooijmanstim]

I notice that DataStreamMonitor compiles its variables (by instantiating DatasetEvaluator) right in its constructor. It would be nice to delay this until before_training to have a tighter debugging cycle, as I often get crashes due to typos etc. between setting up monitoring and the main loop and actually training.

[dwf]

+1

[dmitriy-serdyuk]

On the other hand, @lamblin said to me that it's better to compile your functio as early possible with current version of Theano because in forks a new process for compilation and allocates as much memory as a parent process.

Even if this Theano bug fixed, I don't like that different functions are compiled in different time because it's tricky to measure total compilation time. Should we add a new callback for compilation? It should solve all the problems.

[rizar]

We could do all compilation in before_training, that's where the algorithm compilation is currently done.

I like the proposal, by the way.