Document new GAP gradient algorithm

[max-veit]

Following up from #293, it would be nice to have documentation for the algorithm and the individual steps and equations involved in computing the gradient of a dot-product kernel-based model. It's currently documented on a whiteboard and in a presentation, but we'd like this accessible from the developer documentation as well.

[felixmusil]

I agree that such doc is needed. My concern is the duplication of the information so I would either have a full doc on the subject and reference it in the different parts of the code of have the different parts hold this the bits and pieces. I prefer the former since it allows to gather all relevant info and it should reference the relevant cpp/python bits. However, I don't think we can reference a .rst file that belongs to the doc on the python side in doxygen (cpp side) in an automatic/integrated way. So the back reference would either be static like 'see this part of the doc' or we still incorporate the relevant equations and context leading to some duplications ?

I guess that more than just this algo we should move all the latex explanations about the model to the documentation and reference it in the code. For example I would move the model and sparse cosine kernel explanations to the doc and replace these by 'for more details see sparse GPR model documentation'. Do you agree ?

We can integrate notebooks in the doc and notebooks can render latex so we could also centralize the basic 'how to' of the model in equations and code. Do you like this idea ? Here is an example from Sergey.

[Luthaf]

However, I don't think we can reference a .rst file that belongs to the doc on the python side in doxygen (cpp side) in an automatic/integrated way.

We can do this by using breathe embed:rst capabilities. Basically, this would look like this:

.. equation.rst

.. _emc2:
here is the equation: :math:`e = mc^2`

/// The Equation class, implementing the equation 
/// @verbatim embed:rst 
/// :ref:`emc2`
/// @endverbatim
class Equation {
   // ...
}

It can also be modified to look nicer on the C++ side by adding a new doxygen command as an ALIAS in doxygen config, and having it expand to create a reference in rst.

/// The Equation class, implementing the equation @rst_ref{emc2}
class Equation {
   // ...
}

---

I'm using something similar to include code example in the documentation with this doxygen config.

[max-veit]

This is indeed closer to what I was thinking - document in proper LaTeX close to the code, and have it compile to nice, pretty equations for the documentation. Although that still doesn't remove the need to have a clear, accessible explanation of the algorithm somewhere in the developer docs, since that's (usually) too much to fit in a doc comment.

[felixmusil]

I am not sure to follow what you mean, could you elaborate a bit more what you have in mind ?

[max-veit]

Ok, so we write the equations in the doc comment (the one at the beginning of the function that starts with /**), so that we see them both in the code, close to where they're implemented, and at the same time rendered into pretty math in the Doxygen docs (for instance, here - the LaTeX code is in the C++ source file, close to where it's implemented, and the rendered version is online).

At the same time, it's a good idea to have a somewhat longer, step-by-step explanation in addition to what's in the doc comment (since the comment should be a relatively short summary). So something like what we have with the "SOAP Theory" documentation -- which still needs some work, but anyway -- and with a link to the Doxygen doc page in the appropriate places.