Excessive optimism test

[basnijholt]

(original issue on GitLab)

opened by Anton Akhmerov (@anton-akhmerov) at 2017-12-22T13:15:54.380Z

If the loss is related to the error from the true value (e.g. the error of the integral in cquad),then:

• Eventually the estimated function must lie within the uncertainty from the true value
• If, like in cquad, the learner has a chance of estimating whether the error estimate is reliable, then the estimate must be within the uncertainty from the correct value, except in extraordinary circumstances.