pymc-examples
pymc-examples copied to clipboard
pymc-devs
how to debug a model
File: https://github.com/pymc-devs/pymc-examples/blob/main/examples/pymc3_howto/howto_debugging.ipynb Reviewers:
The sections below may still be pending. If so, the issue is still available, it simply doesn't have specific guidance yet. Please refer to this overview of updates
Known changes needed
Changes listed in this section should all be done at some point in order to get this notebook to a "Best Practices" state. However, these are probably not enough! Make sure to thoroughly review the notebook and search for other updates.
General updates
ArviZ related
Changes for discussion
Changes listed in this section are up for discussion, these are ideas on how to improve the notebook but may not have a clear implementation, or fix some know issue only partially.
General updates
ArviZ related
Notes
Exotic dependencies
Computing requirements
I'd like to take this one. @OriolAbril, you can assign it to me (not sure if that's part of the process). Thanks!
@kuvychko There are a couple of ways to "claim" an issue:
- On the issue, you can say "I am working on this"
- You can type "take" (or "/take") and it should assign it to you.
@kuvychko Sorry to misinform. This works in the scikit-learn repo, so I assumed it was a default GitHub feature.
Ref: https://scikit-learn.org/stable/developers/contributing.html#contributing-code Example:
One easy way to find an issue to work on is by applying the “help wanted” label in your search. This lists all the issues that have been unclaimed so far. In order to claim an issue for yourself, please comment exactly /take on it for the CI to automatically assign the issue to you.
I assume it needs to be set up for this repo.
@reshamas - no worries, thank you!
I started working on this notebook, and ran into an issue. The example used no longer works as intended: with step = pm.Metropolis() sampling fails immediately with a very clear exception message:
SamplingError: Initial evaluation of model at starting point failed!
Starting values:
{'mu': array(0.), 'sd': array(0.)}
Initial evaluation results:
{'mu': -0.92, 'sd': -0.92, 'obs': -inf}
This obviates the need to print out intermediate values. If I use a default NUTS sampler it works just fine even with the wonky priors. Any thoughts on a different example to show use of aesara.printing.Print? Or should this notebook be removed from examples?
There are always examples of broken models on Discourse. I'll try to browse a bit to find one. It is nice to see the error message is so much nicer now that the "debigging for wrong initial point issues" is no longer needed!
I do think the notebook is still needed though, and I am confident we'll find another model to show how to debug, maybe some model that has problematic shapes for example?
Or a good starting point but bad constraints on the parameters (which might even work on the current notebook), what happens for example if you set sd = pm.Normal("sd", mu=1, sigma=1) or with mu=2 instead of 0? Then the starting point should work (IIRC the starting point is the mean, and most values will also work, but the probability of <0 sd is not 0, so we are bound to run into issues anyways I think.
@OriolAbril - indeed, new exception message is excellent, very neat improvement! Thanks for your advice, I'll experiment with bad priors, and keep working through this notebook. If you find a good example please let me know! By any chance, do you know who is the original author of this notebook? It doesn't have that info.
One way printing is useful is if you have intermediate operations between variables, so that you don't know the exact value that is leading to problems.
A perhaps dumb example is a model where the likelihood noise follows a linear combination of the kind sigma=a + b*x, and printing a + b*x reveals zeros or negative values.
@ricardoV94 - thanks, I'll experiment with your example. Question: what if instead of using aesara.printing.Print, you route intermediate values to a Deterministic variable? Seems like both methods should work in a similar way.
The model will likely fail without any way of seeing what the value of the deterministic was.
This is also true when a model fails during sample or when fitting woth variational inference, where there is not such a nice helper output as the one we added for the model initial position when sampling starts
Hi everybody, I ran into an issue trying to commit my new version of the notebook. During pre-commit a copy of the notebook in md format is created in the same folder (examples/howto/howto_debugging.md) instead of myst_nbs/howto/howto_debugging.myst.md, therefore jupytext check is filing. Any advice on how to fix this? Here is a piece of pre-commit output:
[jupytext] Reading examples/howto/howto_debugging.ipynb in format ipynb
[jupytext] Updating the timestamp of examples/howto/howto_debugging.md
does the ipynb file have some custom jupytext metadata that is overriding the project configuration?
I made no changes to the metadata. Here how it looks like for howto_debugging.ipynb (problem notebook):
{
"kernelspec": {
"name": "python3",
"display_name": "Python 3 (ipykernel)",
"language": "python"
},
"jupytext": {
"formats": "ipynb,md:myst"
},
"language_info": {
"name": "python",
"version": "3.10.5",
"mimetype": "text/x-python",
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"pygments_lexer": "ipython3",
"nbconvert_exporter": "python",
"file_extension": ".py"
}
}
and here is the metadata for spline.ipynb (myst.md export worked for it). It has the extra "interpreter" bit and a hash value.
{
"interpreter": {
"hash": "87bd0c5c372351de506bff22b267ffb5c759e6b8276deaa6e4203adc4dc0d335"
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3 (ipykernel)",
"language": "python"
},
"language_info": {
"name": "python",
"version": "3.10.5",
"mimetype": "text/x-python",
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"pygments_lexer": "ipython3",
"nbconvert_exporter": "python",
"file_extension": ".py"
}
}
I don't know what the interpreter thing but I think it is not relevant here. The key part are these lines:
"jupytext": {
"formats": "ipynb,md:myst"
},
which should not be there as they are overriding the project level jupytext config.
@OriolAbril - a question on hyperlinks in notebooks. In splines notebook there are several hyperlinks to externals pages, and jupytext check was happy. In my initial version of "how to debug" I had two hyperlinks, one to Aesara doc and another to github repo of Lindley Lentati. Both properly formatted (same as in passing splines notebook). pre-commit was failing due to presence of these links. Since both links were not critical to the notebook I removed them. but I'd like to understand hyperlink check criteria for my future contributions - can you explain what's the logic there, or point me in the right direction if it's documented somewhere?
I thought we had updated the name of that check. There is nothing inherently wrong with external links, there can be as many external links as you want. The check is against urls that should instead be sphinx references. In this case, the link to the Aesara documentation should not be a url but a sphinx cross-reference instead. I added a section on how to use sphinx-cross references in my pymc-examples webinar material: https://pymc-data-umbrella.xyz/en/latest/about/contributing_to_documentation/docs_presentation.html#cross-referencing-roles.
The reason for this is that sphinx-cross references are much more robust. To generate a link to the api docs for aesara.printing.Print you can use {class}`aesara.printing.Print` which sphinx renders as a url to the correct documentation. If using sphinx cross-references, if the aesara docs move from aesara.readthedocs.io to aesara.ai for example, we will only need to modify one line in conf.py and all links to aesara docs in all notebooks will work again. If the documentation is restructured (like we just did with pymc one for example) changing the specific uls in the API section, regenerating the pymc-examples website alone will be enough to fix all the links. If instead we used urls, any change would require going notebook per notebook, url per url in order to fix them.
This becomes even more relevant for internal cross-references. One can read https://www.pymc.io/projects/examples/en/latest (default, to see the lasest state of all notebooks) or https://www.pymc.io/projects/examples/en/2022.01.0 (frozen snapshot of the website, to see what was the state of the example gallery in January, when most notebooks run on v3 for example). If we use urls for these kind of references, it is impossible for our own website to be coherent. The url hardcodes the version on which to link, the sphinx cross-references do not and will always link to the version being read; to put an example, if the spline notebook links to the debugging one, and uses sphinx cross-references, the version of the example gallery I am reading won't change under my feet inadvertedly when I click on the link.
@OriolAbril - really appreciate the details, thanks!!! sphinx write-up is really helpful, too. I didn't realize that aesara falls under the same rubric as pymc in terms of documentation, which makes total sense! looks like a well thought-out system.
You can see in the pre-commit config which are the websites whose urls are "forbidden": https://github.com/pymc-devs/pymc-examples/blob/main/.pre-commit-config.yaml#L87-L97