[tests] JavaScript: extract searchindex.js-format test fixtures.

[jayaddison]

Feature or Bugfix

• Refactoring

Purpose

• Reformats the test index data in the JavaScript HTML search tests to match the format emitted by the Sphinx (Python) HTML builder.

Detail

• The content of searchindex.js is a JavaScript code file that runs Search.setIndex(...) with the contents of the index as an argument.
• Instead of loading the index per-test from a variable, retrieve each fixture from the built-in local karma HTTP server and eval it.
• Includes fixtures generated from genuine, buildable test Sphinx project sources included within this repository.
• Fails the Python test suite when regenerated searchindex.js fixture content does not match the existing fixture files.

Relates

• Resolves #12099.

[jayaddison]

Another challenge: the C++ term in an existing test index needs to be replicated if we follow this approach. The current EN-language stemmer maps the input term C++ to c (lowercase).

I don't want to merge this PR until determining a path forward to resolve that, because there might be a conflict otherwise.

Using a null term stemmer could be a solution, but I'd have to figure out how to do that.

[jayaddison]

In follow-up pull requests, the plan is to generate fixtures from genuine, buildable test Sphinx project sources included within this repository.

This turned out to be less code than I expected, so I've bundled it in here too, but could separate it again if that's easier for review/understanding.

[jayaddison]

@picnixz when you have time, could you review this pull request next, and then #12107 after that?

I know there are a lot of these small changes / PRs, so it's probably tricky to develop the context for what's changing. Basically: with these two I want to make it much easier and better to write accurate test coverage, by using 'genuine' generated Sphinx indexes, and using the proper JS query+search code in the tests.

I think it's important enough to block the other search-related PRs until these are in.

[picnixz]

I know there are a lot of these small changes / PRs, so it's probably tricky to develop the context for what's changing.

Which is the reverse of what I'm doing on the test suite where the PRs are huge. So don't worry

[jayaddison]

The latest changes in mainline make a small change to the searchindex.js file format. I'm going to apply a change here too (including merging in a utility script that I've written to perform the regeneration and beautification).

[wlach]

@jayaddison is there anything more that needs to happen with this one? anything I can do to help get it over line?

[jayaddison]

Thanks @wlach! I think I should re-test this manually after the latest merge; any additional code review feedback would be useful. I do remember that there was some JavaScript variable assignment logic that I felt was a bit non-obvious (see the within-line comments attempting to explain it).

[jayaddison]

Thanks @wlach! I think I should re-test this manually after the latest merge; any additional code review feedback would be useful. I do remember that there was some JavaScript variable assignment logic that I felt was a bit non-obvious (see the within-line comments attempting to explain it).

Ok, yep - manually testing from 9f5df2cccaf41e81b58fee9a47bf98edfbe798cf seems good here; I took the opportunity to merge/re-test #12047 after doing that, and it tests correctly too (the Welcome search test case for sphinx.git de-duplicates two top search results into one with the fix in place; the search results for configuration continue to be a bit spammy, but that doesn't appear to be due to title-based duplication).

[jayaddison]

Thank you @wlach for the code review! I've left a couple of not-entirely-resolved items open, and let me know what you think of the adjustments.

[wlach]

@picnixz Could you take a look at this? I think it makes sense to merge this before proceeding with any more work on the search system. I reviewed it a couple weeks ago and all looks good to me (not to say I'm infallible but that's gotta count for something)

[picnixz]

Could you take a look at this? I think it makes sense to merge this before proceeding with any more work on the search system

Lucky you that I've submitted my thesis and have more free time!

[picnixz]

Once you're done, just ping me and I'll merge it !

[jayaddison]

Thanks @picnixz - all done with those changes, so I think this is ready for merge.

[picnixz]

Thank you!