Make some dependency changes, support Python 3.10

[Jackenmen]

Description of the changes

This PR makes a bunch of dependency changes (not just updates) and enables support for Python 3.10.

Added primary dependencies:

• typing_extensions (we don't use it but we probably will in the future)
• yarl (we use it in our code so it shouldn't be transitive)
• aiodns, Brotli, orjson (some of the speedup dependencies for aiohttp and discord.py)

Replaced dependencies:

• apsw-wheels with apsw
• appdirs replaced by platformdirs
  • I'm still not entirely sure this should happen, a lot of popular projects (pip, virtualenv, pipenv, black, isort) have switched to it which suggests it's going to be preferred in future but I have seen a few worrying things about how maintainer approaches breaking changes. Not that I don't go through changelogs of all our dependencies but I'm not a fan of this.
• chardet replaced by charset-normalizer (transitive dependency of aiohttp)
• fuzzywuzzy replaced by rapidfuzz

Removed dependencies:

• aiosqlite (this library is not used by Red anymore)
• PyNaCl (transitive dependency of discord.py[voice] which we don't actually use)
• python-Levenshtein-wheels (transitive dependency of fuzzywuzzy which we replaced with rapidfuzz)

The PyNaCl removal is still up for consideration, I'm not entirely sure why we have it added if we're using LL rather than d.py's own voice functionality.

Test run of updated install instructions: https://cirrus-ci.com/build/5878059449450496

Have the changes in this PR been tested?

No

[aikaterna]

I'm using a fork for testing some things that includes this PR, and I received this error with a brand new Python 3.10 install (installed through Brew) with a brand new venv. This is also a new Red install inside that venv. I had existing instances already configured on this device with alternate python installs and alternate venvs.

(hybrid) aikaterna@Mac-mini ~ % redbot-setup
Traceback (most recent call last):
  File "/Users/aikaterna/hybrid/bin/redbot-setup", line 5, in <module>
    from redbot.setup import run_cli
  File "/Users/aikaterna/hybrid/lib/python3.10/site-packages/redbot/setup.py", line 17, in <module>
    from redbot.core.cli import confirm
  File "/Users/aikaterna/hybrid/lib/python3.10/site-packages/redbot/core/__init__.py", line 4, in <module>
    from .config import Config
  File "/Users/aikaterna/hybrid/lib/python3.10/site-packages/redbot/core/config.py", line 22, in <module>
    from .drivers import IdentifierData, get_driver, ConfigCategory, BaseDriver
  File "/Users/aikaterna/hybrid/lib/python3.10/site-packages/redbot/core/drivers/__init__.py", line 4, in <module>
    from .. import data_manager
  File "/Users/aikaterna/hybrid/lib/python3.10/site-packages/redbot/core/data_manager.py", line 61, in <module>
    _old_config_location.rename(config_file)
  File "/opt/homebrew/Cellar/[email protected]/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pathlib.py", line 1232, in rename
    self._accessor.rename(self, target)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/aikaterna/Library/Application Support/Red-DiscordBot/config.json' -> '/Users/aikaterna/Library/Preferences/Red-DiscordBot/config.json'

I copied my config.json from the Application Support directory to the Preferences directory and I was able to use redbot-setup.

[Jackenmen]

@aikaterna thanks for testing! This should be resolved now.

[aikaterna]

I encountered the same issue on another Mac that had a Python 3.9 venv, while running redbot and not redbot-setup but using the changes in https://github.com/Cog-Creators/Red-DiscordBot/pull/5611/commits/7e740624382745c1e2a5198bab426e109abd4852 has resolved it. Posting the traceback for information only, in case you were interested.

(redenv39) username@iMac ~ % redbot margaret --dev
Traceback (most recent call last):
  File "/Users/username/redenv39/bin/redbot", line 5, in <module>
    from redbot.__main__ import main
  File "/Users/username/redenv39/lib/python3.9/site-packages/redbot/__main__.py", line 28, in <module>
    from redbot.core.bot import Red, ExitCodes, _NoOwnerSet
  File "/Users/username/redenv39/lib/python3.9/site-packages/redbot/core/__init__.py", line 4, in <module>
    from .config import Config
  File "/Users/username/redenv39/lib/python3.9/site-packages/redbot/core/config.py", line 22, in <module>
    from .drivers import IdentifierData, get_driver, ConfigCategory, BaseDriver
  File "/Users/username/redenv39/lib/python3.9/site-packages/redbot/core/drivers/__init__.py", line 4, in <module>
    from .. import data_manager
  File "/Users/username/redenv39/lib/python3.9/site-packages/redbot/core/data_manager.py", line 62, in <module>
    _old_config_location.rename(config_file)
  File "/usr/local/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pathlib.py", line 1382, in rename
    self._accessor.rename(self, target)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/username/Library/Application Support/Red-DiscordBot/config.json' -> '/Users/username/Library/Preferences/Red-DiscordBot/config.json'

[Zephyrkul]

Getting this traceback in my console whenever I get a CommandNotFound when I try this PR out:

Ignoring exception in on_command_error
Traceback (most recent call last):
  File "C:\Users\...\Desktop\Github\Red-DiscordBot\.venv\lib\site-packages\discord\client.py", line 375, in _run_event
    await coro(*args, **kwargs)
  File "C:\Users\...\Desktop\GitHub\Red-DiscordBot\redbot\core\events.py", line 260, in on_command_error
    fuzzy_commands = await fuzzy_command_search(
  File "C:\Users\...\Desktop\GitHub\Red-DiscordBot\redbot\core\utils\_internal_utils.py", line 159, in fuzzy_command_search
    extracted = process.extract(term, choices, limit=5, scorer=fuzz.QRatio)
  File "cpp_process.pyx", line 945, in rapidfuzz.cpp_process.extract
  File "cpp_process.pyx", line 803, in rapidfuzz.cpp_process.extract_list
  File "cpp_process.pyx", line 745, in rapidfuzz.cpp_process.extract_list_f64
  File "cpp_process.pyx", line 166, in rapidfuzz.cpp_process.preprocess_list
  File "cpp_utils.pyx", line 44, in rapidfuzz.cpp_utils.default_process_capi
  File "cpp_common.pxd", line 201, in cpp_common.conv_sequence
  File "cpp_common.pxd", line 171, in cpp_common.hash_sequence
TypeError: object of type 'Command' has no len()

[aikaterna]

I usually like to be a little more prepared with tracebacks and such when making a report like this, but I haven't been able to test the conditions fully so I thought I'd leave a note here instead. When I was installing this PR on a device that is Linux aarch64, but that is not supported by Red, I was unable to compile rapidfuzz==2.0.8 due to some compilation changes they made in the 2.x.x series, and had to use/install rapidfuzz==1.9.1 instead to get things to compile. This may not apply or may not be an issue at all on other supported OSes on aarch64, but it may be good to test this dep/PR on a system that qualifies as that designation to make sure there's no issues.

[leetfin]

@aikaterna I was able to run this just fine on my Linux aarch64 without needing to downgrade dependencies - let me know if you want me to test anything.

[aikaterna]

@aikaterna I was able to run this just fine on my Linux aarch64 without needing to downgrade dependencies - let me know if you want me to test anything.

That's great to hear, thanks for commenting. If everything worked properly in the regular install/build process then that's all I was concerned about.

[leetfin]

I ran it on this branch of my fork, but I didn't change anything about rapidfuzz.

[aikaterna]

I was able to test installing this on a 32 bit Raspbian OS 11 install and it encountered build issues with ninja 1.10.2.3 which is the last pypi published version. I git cloned their newest release on their github @ 1.11 and I was able to build it with their python wrapper script with no issues (except for a couple warnings) although I couldn't get it to link properly/have this install use ninja 1.11 instead. Attached are the logs from the standard install process in case it provides any more pertinent info for this situation.

https://gist.githubusercontent.com/aikaterna/2b056087e34df2f31e5fe750c3673713/raw/5830661c0b830d007cff9a8d680c61aec6edf38a/raspbian_11_build.txt

https://gist.githubusercontent.com/aikaterna/b470fad8cef535d956475a75c3b6c2f8/raw/df19a98e047a9d7514ed2b1c213968756ee70730/ninja_1-11_build.txt

[Jackenmen]

Ideally, we should switch from apsw-wheels to apsw PyPI package in this PR as well.

Right now there are no wheels for manylinux aarch64 (which is going to be fixed in next apsw release, whenever that happens) and macOS arm64 (I'm unaware of any plans for adding this right now). Before I do such change, I'll have to test that these projects are buildable from source on those platforms as well as on RPi OS 10 armv7l which can't use piwheels due to a non-default Python version being used there.

[leetfin]

Adding a note here - the requirements in this PR are one build behind on Red-Lavalink, which will break audio compatibility with discord.py v2.

[leetfin]

Are any PRs aside from #5924 blocking this?

[Jackenmen]

Are any PRs aside from #5924 blocking this?

Until today, pycares release soft-blocked this PR as there was no release with 3.11 wheels for macOS arm64. That has been addressed now though.

Ideally, I would like to wait for Python 3.11.1 to be available on Chocolatey. However, this at the very least depends on:

• the merge of this PR: https://github.com/chocolatey-community/chocolatey-packages/pull/2075
• approval of this package after the above PR is merged: https://community.chocolatey.org/packages/python311

It may take a while so we might need to just stick to 3.10.8 on Windows for now though. I don't want to use Python 3.11.0 as usually a lot of issues found in RC versions get deferred to .1 release.

Other than that, I'll be going through changelogs of our dependencies probably sometime next week to see if there are any breaking changes I might have not accounted for already as I lost track of when was the last time I looked through them.

[Jackenmen]

Ready for review. Install instructions tests pass in CI, I looked through changelogs of the updated dependencies, as well as the deprecations and removals in Python 3.10 and 3.11 changelogs.

Things to look out for during the review:

• errors (obviously)
• new deprecation warnings from Python 3.10 and 3.11 (ideally this PR should be tested on both 3.8 and 3.11)
• when running on a macOS computer that had any instance set up on an earlier Red version (any version before this PR), the metadata of that instance should still be detected properly by Red
• code that used to use fuzzywuzzy but now uses rapidfuzz
  • the fuzzy results don't have to be identical but the new code should still behave in the same way

---

Note that the changes related to the import system (e.g. <frozen importlib._bootstrap>:283: DeprecationWarning: the load_module() method is deprecated and slated for removal in Python 3.12; use exec_module() instead) will come in a separate PR.

[Flame442]

Did some basic testing of stuff I usually do with Red on 3.8.5 with this PR, seems like everything is still working as I expect it to.

[PredaaA]

It works without any error on macOS on both 3.8 and 3.11.