Add examples to API documentation

[alliesaizan]

Related to issue #730, this issue covers adding links to relevant example notebooks to the API documentation. The fix would be for someone to go through the notebooks and add links in the API documentation for the classes/functions used in the example. Tagging @hildeweerts and @romanlutz in case they have additional ideas!

[yanhong-zhao-ef]

would love to have a go at this if it is not covered already

[hildeweerts]

For whoever will work on this, note @adrinjalali's suggestion in this comment:

One way to fix the original issue is to automatically have links to all examples from the API guides the same way scikit-learn does (example: https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html#examples-using-sklearn-compose-columntransformer)

[romanlutz]

Update: @yanhong-zhao-ef also made a useful addition, but that isn't quite what this issue asks for so we'll leave it open and it's up for grabs for anyone!

The latest task to do here is to find out how sklearn is getting the "gallery" of examples into the API reference. That would automatically tackle this problem. I believe this configuration is responsible: https://sphinx-gallery.github.io/dev/configuration.html#references-to-examples

So someone would need to try this (using the website build) and can check with how sklearn is doing it as a comparison, and validate that it works.

[miacodehub]

hi! would love to work on this if it is still available

[romanlutz]

Absolutely! Do you know what to do?

[miacodehub]

I believe I understand. Correct me if I'm wrong: Currently there's one example notebook for each of the APIs documented. In the API reference pages, at the bottom of the page there should be a section with title "examples that use API #1" followed by the "gallery" representation to that example. Since there's currently five APIs with five example notebooks, each API reference page will have its resepective example notebook. I'll follow the suggestion you mentioned w.r.t the config file modifications. Do let me know if I missed anything.

[alliesaizan]

I believe I understand. Correct me if I'm wrong: Currently there's one example notebook for each of the APIs documented. In the API reference pages, at the bottom of the page there should be a section with title "examples that use API #1" followed by the "gallery" representation to that example. Since there's currently five APIs with five example notebooks, each API reference page will have its resepective example notebook. I'll follow the suggestion you mentioned w.r.t the config file modifications. Do let me know if I missed anything.

Here's an updated link to the example notebooks page btw: https://fairlearn.org/v0.10/auto_examples/index.html. So there isn't a 1:1 mapping between the example notebooks and the API reference. The API reference has 5 modules: datasets, preprocessing, postprocessing, metrics, reductions. These example notebooks each use a specific method/class/function within their respective module, so the links should go in the API reference section corresponding to that method(s).

The way we might be able to accomplish this task is to use sphinx to replicate the scikit-learn documentation. For instance, in the RandomForestClassifier API documentation, there's a gallery at the bottom of the page titled "Examples using [method]". We'd want to copy this format in our API reference, where, for example, the MetricFrame API page has a gallery of all the notebooks that use the MetricFrame class. The link Roman shared provides some information on how to do this, but we will want to get this working for our documentation. I would suggest starting there, building the docs locally, updating the configurations, and let us know what you find. Feel free to post any issues you run into as you work on this. I know Sphinx can be tricky to work with so I am happy to provide pointers or look into things if you get stuck!

[miacodehub]

Thank you for clarifying @alliesaizan ! I have gone through the configuration that scikit uses to link sphinx gallery. I'll try to emulate that and see if it works.

[miacodehub]

Hi all! I followed the steps for building the html pages and turned on "examples using.." following what scikit does and the changes in the conf.py suggested in sphinx gallery documentation but I am not able to see these changes. I installed the required frameworks and packages as well.

I have a few questions w.r.t the issues I'm seeing:

1. I am getting a few warnings like the below example while building the docs using the "python -m sphinx -v -b html -n -j auto docs docs/_build/html" command. Should these be eliminated?

C:\Users\RamyaMahavratayajula\fairlearn\fairlearn\adversarial_adversarial_mitigation.py:docstring of fairlearn.adversarial._adversarial_mitigation.AdversarialFairnessClassifier:1: WARNING: py:class reference target not found: BackendEngine C:\Users\RamyaMahavratayajula\fairlearn\fairlearn\adversarial_adversarial_mitigation.py:docstring of fairlearn.adversarial._adversarial_mitigation.AdversarialFairnessClassifier:1: WARNING: py:class reference target not found: default = 'auto'

2. In the Sphinx guidelines, they suggest changes in the conf.py file as follows: sphinx_gallery_conf = { 'backreferences_dir' : 'gen_modules/backreferences',

   'doc_module' : ('sphinx_gallery', 'numpy'),

   'exclude_implicit_doc': {r'pyplot.show'}, } I was looking at scikit's documentation and did some research to have the values for fairlearn as follows: doc_module : "fairlearn" backreferences_dir : "example_ref" // doc_module just had to be filled with the module name for which documentation has to be regenerated. backreferences_dir was a little more confusing. Is it supposed to be a new/existing location to store the newly generated galleries? or should it point to existing gallery reference locations - for example "api_reference" ? I tried both these but didn't get any results.

In addition to these changes, are there any other places I should be looking at to make changes?

Any advice/suggestion would be incredibly helpful. Thank you.

[romanlutz]

Do you have a branch that you can share?

[miacodehub]

Yup. My branch name is "miacodehub". I created a pull request with the changes I made. Unfortunately, I'm still unable to see the galleries in API docs. I made the following changes in the conf.py:

• added module names to the doc_module attribute
• created a new folder "backref_dir" to hold the generated rst files for the modules

Also, the examples for scikit-learn are structured into different folders based on their modules, if that makes any difference. It wasn't mentioned anywhere in the scikit/sphinx documentation if that was necessary so I didn't look further into it.

Do share if there's anything else I can do.

[alliesaizan]

Yup. My branch name is "miacodehub". I created a pull request with the changes I made. Unfortunately, I'm still unable to see the galleries in API docs. I made the following changes in the conf.py:

* added module names to the doc_module attribute

* created a new folder "backref_dir" to hold the generated rst files for the modules

Also, the examples for scikit-learn are structured into different folders based on their modules, if that makes any difference. It wasn't mentioned anywhere in the scikit/sphinx documentation if that was necessary so I didn't look further into it.

Do share if there's anything else I can do.

I can see the branch on your fork, but not the PR in the fairlearn repo. Could you let me know if you're sure you opened it there? I think you're on the right track, but I have some recommendations about how to alter conf.py further. It would be easier for me to reference the conf.py file directly in a PR. When you built the documentation, I am guessing it looked the same as before you made the changes? If it changed, it'd be useful to see those changes too (via screenshots of the generated HTML files).

[miacodehub]

Hi! So I created a new branch called "miacode" and created a pull request. You are right in that there's no change after rebuilding documentation.

[alliesaizan]

I see it now, thank you!