Andrew Lavin

@soumith thank you for continuing to lead this valuable benchmark project, which benefits the entire machine learning community. +1 for Intel GPUs. They are potentially very relevant for inference. In...

@soumith Are accuracy / convergence tests back on the table? Per layer numeric accuracy and iterations to / accuracy of convergence are both informative measures. We do not really understand...

@soumith I have been a software engineer for too long to believe that good intentions are enough to ensure accuracy. ;-) I do believe that rewarding frameworks for speed without...

@scott-gray I am sure you know the difference between an idea that seems like it should work and an experimental result that is publicly available. > In order to believe...

While I find the discussion of standardization interesting and important, I really do not need any of that to do basic convergence testing. For example, if I had Neon configuration...

@vrv Convergence speed vs. accuracy tests _are_ benchmarks. But the idea has no traction here so I will do it somewhere else.

@soumith yes convergence tests of course require a lot of compute time. can we at least incorporate numeric accuracy tests with synthetic data, as I think @vrv mentioned earlier?

The approach I would take is: A) kernel efficiency versus numeric accuracy testing B) whole network convergence A/B testing (vary one of 1. kernel, 2. precision, 3. batch size, etc...)....

The 3 different benchmarks I outlined A) kernel speed/accuracy, B) network convergence, C) framework profiling are independent of each other, can be done in any order, and can even be...

@hughperkins please read my posts more carefully. I wrote that deepmark could be a framework profiling tool rather than just a forward/backward iteration timing tool. TensorFlow is now instrumented for...