Removing JavaScript Dependencies

[TimKam]

Relates to https://github.com/sphinx-doc/sphinx/pull/10028. Sphinx plans to remove jQuery and underscore JS dependencies in version 6.0.0 (i.e., eventually, there is plenty of time). This will probably break many extensions and themes and we need to try and find at least the most important ones.

TODOs:

• [ ] Remove 'internal' usage of dependencies: https://github.com/sphinx-doc/sphinx/pull/10028
• [ ] Fix dependency usage of all extensions hosted by https://github.com/sphinx-doc and https://github.com/sphinx-contrib/ in some way (either refactoring or including dependencies, but ideally refactoring).
• [ ] Identify as many of the frequently used 3rd-party themes and extensions as possible and warn the maintainers.
• [ ] Remove dependencies
• [ ] Anything else?

[TimKam]

Looping in some Read the Docs folks: @ericholscher & @nienn: Do you have any advice or can you provide any intel based on RTD usage? For example, we could check the most important extensions and themes, based on RTD usage stats. Thank you.

[AA-Turner]

From a quick look, the only usage in https://github.com/sphinx-doc I can see [^1] is https://github.com/sphinx-doc/sphinxcontrib-websupport/blob/master/sphinxcontrib/websupport/files/websupport.js

Popular/impactful themes -- I'd probably suggest looking at at least:

• https://github.com/python/python-docs-theme
• https://github.com/pydata/pydata-sphinx-theme
• https://github.com/pradyunsg/furo (not impacted https://github.com/sphinx-doc/sphinx/issues/10070#issuecomment-1008133964)
• https://github.com/readthedocs/sphinx_rtd_theme
• https://github.com/bashtage/sphinx-material/
• https://github.com/executablebooks/sphinx-book-theme

Actual data would of course be more helpful!

[^1]: This is the link for GitHub's new code search preview -- I also went through for underscore and the $ and $u symbols. https://cs.github.com/?scopeName=All+repos&scope=&q=org%3Asphinx-doc++jQuery

A

[pradyunsg]

I can confirm that Furo doesn't depend on jQuery or UnderscoreJS.

[TimKam]

Thanks! I forgot that there is a separate org for many extensions: https://github.com/sphinx-contrib/ (I do not have access to the new GitHub code search feature, yet.)

[pradyunsg]

Doing a search on sphinx-contrib brings up references in findanything and images.

There's a total of 10 results, all in these two. Okay, there's one in zopeext but that's in an HTML file.

[TimKam]

I now have access to GitHub code search and found these two extensions that use jQuery:

• https://github.com/sphinx-contrib/cheeseshop
• https://github.com/sphinx-contrib/findanything I'd assume they are not widely used.

[ericholscher]

@TimKam Sadly we don't have great stats on extension usage across the platform. This is something we hope to figure out eventually, but not something we have currently.

[TimKam]

Thanks, @ericholscher. Let us know if there's anything we need to consider from RTD's perspective when removing jQuery and underscore. @tk0miya I think many templates will break if we don't announce this carefully. Should we add an info message when building or something similar?

[tk0miya]

I don't have good idea to let developers know the deprecation. But info or message warnings on build would not help it, I think. So it would be better to announce it on CHANGES file and sphinx-users group.

[AA-Turner]

An idea I had is (attempting to) patch the actual jQuery object itself in JS, and warn on any method access. I don't think it's great though, as it relies on users checking the JS console.

Agree with @tk0miya on adding notes to CHANGES etc -- my initial PR at #10028 does this, although maybe it could be called out more?

A

[Daltz333]

https://github.com/wpilibsuite provides several popular extensions and we are tracking this change. Not sure when we'll make the change, but we'll get to it.

[marxin]

I've noticed Search results are quite broken, one can see it also on the official website:

https://www.sphinx-doc.org/en/master/search.html?q=--keep-going

Search Results
Your search did not match any documents. Please make sure that all words are spelled correctly and that you've selected enough categories.

Search Results
Your search did not match any documents. Please make sure that all words are spelled correctly and that you've selected enough categories.

[object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object]

I see the section Search Results twice and the [object Object] seems to be a weird result.

What was displayed before: https://sphinx--10240.org.readthedocs.build/en/10240/search.html?q=--keep-going

Search Results
Search finished, found 2 page(s) matching the search query.

sphinx-build.--keep-going (program option, in sphinx-build)
sphinx-build.--keep-going (program option, in sphinx-build)

Moreover, the following is also bad: https://www.sphinx-doc.org/en/master/search.html?q=sphinx-build

...
sphinx-build.11 (program option, in sphinx-build)
sphinx-build.11 (program option, in sphinx-build)
sphinx-build.12 (program option, in sphinx-build)
sphinx-build.12 (program option, in sphinx-build)
sphinx-build.13 (program option, in sphinx-build)
sphinx-build.13 (program option, in sphinx-build)
sphinx-build.14 (program option, in sphinx-build)
sphinx-build.14 (program option, in sphinx-build)
sphinx-build.15 (program option, in sphinx-build)
sphinx-build.15 (program option, in sphinx-build)
...

[AA-Turner]

@marxin looking at the console output at least part of the problem is that read the docs' overwriting of Sphinx search isn't fixed yet -- does the problem persist on 'pure' Sphinx?

A

[marxin]

@marxin looking at the console output at least part of the problem is that read the docs' overwriting of Sphinx search isn't fixed yet -- does the problem persist on 'pure' Sphinx?

Dunno. Is there a bug report about the RTD issue, please?

[marxin]

And I have a project where it fails with master branch + #10153 fix:

https://splichal.eu/scripts/sphinx/demo/_build/html/search.html?q=-Wno-shift-overflow&check_keywords=yes&area=default

[stsewd]

@marxin I'm trying to fix all things related to search on latest sphinx and rtd, I'm tracking the progress at https://github.com/readthedocs/readthedocs.org/issues/8871.

[AA-Turner]

And I have a project where it fails...

https://splichal.eu/scripts/sphinx/demo/_build/html/search.html?q=-Wno-shift-overflow

In that project, https://splichal.eu/scripts/sphinx/demo/_build/html/searchindex.js doesn't have wno-shift-overflow composited in "terms" so the regex match won't work. Do you have a link to the same project on version 4.y.z?

Sorry to ask for more things, I'm just trying to find where the error is -- the stemmer, the index, searchtools.js itself. etc.

A

[marxin]

In that project, https://splichal.eu/scripts/sphinx/demo/_build/html/searchindex.js doesn't have wno-shift-overflow composited in "terms" so the regex match won't work. Do you have a link to the same project on version 4.y.z?

Sure, there's a version built with 4.x branch: https://splichal.eu/tmp/_build/html/search.html?q=-Wno-shift-overflow

[AA-Turner]

@marxin does #10247 fix the issue?

A

[marxin]

@marxin does #10247 fix the issue?

It's better, but still not ideal: https://splichal.eu/tmp/_build2/html/search.html?q=-Wno-shift-overflow

Note 4.x branch lives here: https://splichal.eu/tmp/_build/html/search.html?q=-Wno-shift-overflow

[marxin]

Note you can use the following Demo project for testing: https://github.com/marxin/sphinx-demo

[marxin]

@AA-Turner Can you please take a look at the attached demo project and investigate what's wrong? Thanks.

[AA-Turner]

I was able to fix this I believe @marxin, I unexpectedly had some free time today. Please may you test? I won't be able to reply to comments as quickly as I usually do, sorry. (Edit: see PR #10273)

A

[marxin]

I was able to fix this I believe @marxin, I unexpectedly had some free time today. Please may you test? I won't be able to reply to comments as quickly as I usually do, sorry. (Edit: see PR #10273)

It works fine, thank you.

[humitos]

@TimKam Sadly we don't have great stats on extension usage across the platform. This is something we hope to figure out eventually, but not something we have currently.

@ericholscher

We can get out some data from our database now (https://github.com/readthedocs/readthedocs.org/pull/9113). I did a quick query to check if it gives us the data we are looking for and it did work great:

In [15]: BuildData.objects.filter(
    ...:     data__packages__pip__all__contains=[{"name": "sphinxcontrib-spelling"}]
    ...: ).values("data__project__slug").distinct().count()
Out[15]: 132

We have 132 projects using sphinxcontrib-spelling (I took the first one from https://github.com/orgs/sphinx-contrib/repositories as example).

I'd like to do a __contains together with a __regex for the name, so I can do sphinxcontrib-*, but I think that's not possible. I've tried executing some raw queries using PostgreSQL JSON fields syntax without success 😭

So, the data is there, we need to know the right query we have to execute to find some useful numbers.

[jdillard]

I don't know how accurate the third party PyPi stat websites are, but they can provide download numbers for each package. Example: https://pepy.tech/project/sphinxcontrib-spelling

[pradyunsg]

They're fairly reliable -- they get the data from PyPI's BigQuery downloads table: https://warehouse.pypa.io/api-reference/bigquery-datasets.html

[AA-Turner]

What the stats don't (can't, I imagine) give us is what is pulling each extension in (unique projects) -- it does provide a roughh heuristic though for prioritisation. Sphinx's master branch now has JS frameworks removed, so should be a useful target to test against for the various sphinxcontrib-* projects.

A

[humitos]

@humitos

I'd like to do a __contains together with a __regex for the name, so I can do sphinxcontrib-*, but I think that's not possible. I've tried executing some raw queries using PostgreSQL JSON fields syntax without success sob

I went ahead and did this manually in a dirty way to, at least, have these numbers 😄

1. get the list of all repositories from sphinx-contrib GH organization

$ gh api --paginate -H "Accept: application/vnd.github+json" /orgs/sphinx-contrib/repos | jq -r '.[].name' | sort > sphinx-crontib.repositories

2. query the database for each of them prefixing sphinxcontrib-
3. results: https://gist.github.com/humitos/489b9159b21faffd9ce1d0d2c91ec5ff

Note this data belongs to projects that were built at least once starting in mid-May (when we started collecting this data) and the repository name does not always match the PyPI package name.

I hope this data is useful to make better decisions.

[humitos]

Hi all! I've been working on analyzing some data we have been collecting on Read the Docs after building the "telemetry" database. That data helps us to answer questions like "What are the most used Sphinx extensions in a period?". I wrote a blog about this at https://blog.readthedocs.com/knowing-more-about-ourselves/ and added this particular plot on it.

I think that plot/data could be useful when prioritizing the work required to remove JavaScript dependency and on what extensions work first. Let me know if you find this useful and if you are interested in a particular question that can be answered with the data we have now. Thanks!