Adding tests for rapid inspection of visual elements

[kchawla-pi]

What would you like changed/added and why?

Quoting @jeromedockes here, from https://github.com/nilearn/nilearn/pull/2191#issuecomment-546278888 Since we are adding reports etc. we should add more meaningful inspection of generated plots and reports in the tests

What would be the benefit? Does the change make something easier to use?

I think CircleCI does all this but it takes way too long. Maybe a faster simpler test to quickly generate plots and reports to visually examine will be useful?

[jeromedockes]

I think CircleCI does all this but it takes way too long. Maybe a faster simpler test to quickly generate plots and reports to visually examine will be useful?

Not really, I meant that the tests are at the moment ad-hock and superficial

[kchawla-pi]

You mean something like Automated Visual Inspection for example what Selenium does for the Web Frontend? WIth more detailed reports/plots for testing?

[jeromedockes]

I think you can close this one for now

[emdupre]

One short term solution that gets to this idea -- we could create a reviewing template, such as the one used by fMRIPrep that includes specific pages of the documentation that should be visually inspected !

[Remi-Gau]

Reading this I realize that our doc has no examples of what the reports look like. Am it right?

Maybe instead of having fully automated visual tests à la selenium, we could add examples on the doc of the different reports that nilearn can generate.

This would at least allow us to inspect them manually like any other part of the doc and also to point users to them.

Ideally we could have the examples generate the reports to integrate in the doc.

[ymzayek]

@Remi-Gau can you be more specific? Sorry if I misunderstand but there are calls to maskers' generate_report in some examples and in the user guide. It's true that for GLM generate_report is never called in an example but instead make_glm_report is showcased, which is the function called by glm.generate_report. Do you mean to centralize this info and/or be more explicit about the expected outcome? Btw should we favor calling glm.generate_report over make_glm_report where possible?

[Remi-Gau]

I admit I had more the GLM report in mind.

I think I had more in mind a specific section in the docs for reports only and where each report is show cased.

At the moment even we as maintainers have to remember in which examples those reports are generated to make sure there is no "regression" when we check things manually.

Having a place where they are centralized could help for us but could also give the visual reports more "visibility" (pun intended) to. New users.

Am kind of inspired by mriqc and fmriprep that a link to HTML page containing one example report.

https://mriqc.readthedocs.io/en/latest/reports.html

https://fmriprep.org/en/stable/outputs.html#visual-reports

Does this make sense?

[Remi-Gau]

This may also either reuse the visualization script mentioned in #3665.

Or make it obsolete?

[ymzayek]

Thanks @Remi-Gau yes makes sense. I'm for it