Recursion bug in python3.6 and gevent.monkey.patch_ssl()

[rossengeorgiev]

Continuation from #95

Only in python 3.6 and when using SteamClient, which uses gevent.

Happens when the gevent monkey patches are applied after importing requests module. gevent might have a fix in a future release

code to reproduce

import requests
import gevent.monkey
gevent.monkey.patch_ssl()
requests.get("https://google.com")

Work around

Do the gevent monkey patch before any other code or imports. So at the very top:

import gevent.monkey
gevent.monkey.patch_socket()
gevent.monkey.patch_ssl()

[dvf]

Can confirm that this affects python3.8 too.

Also, the workaround is problematic if you're deploying in Lambdas and using Serverless Framework as they init SSL before you have an opportunity to monkey patch. Any help would be appreciated.

[DevilXD]

I've just ran into this issue myself, and tracking it down was quite a deep rabbit hole. Since it looks like this issue is still open, I though I'd chime in with my two cents on how it affects my application and what my mitigation was.

For context, my main application is https://github.com/Cog-Creators/Red-DiscordBot, which is a modular asyncio and uvloop based Discord bot, that supports expanding it's functionality via 3rd party modules created by the Red's community. Most of my personal use modules are self-written, but I still use a few of them that come from the community itself. Now, I'm aware that asyncio and gevent don't really work together on it's own, but I've managed to get around this like so:

    def _fetch_PTS_version(self) -> Optional[str]:
        client = SteamClient()
        if client.login(*self._steam_creds) != EResult.OK:
            return None
        cdn = CDNClient(client)
        version_file = next(
            (cdn_file for cdn_file in cdn.iter_files(
                596350, "Binaries\\GameVersion.txt", filter_func=lambda d, i: d == 596351
            )),
            None,
        )
        if version_file is None:
            return None
        return version_file.read().decode().strip()

    async def fetch_PTS_version(self) -> Optional[str]:
        return await self.bot.loop.run_in_executor(None, self._fetch_PTS_version)

This is literally all of my code that uses the steam package. Now, about 2 days ago, I've also ran into an issue of my bot "dying" out of nowhere and throwing weird SSL recursion error tracebacks, which I've only today managed to track down to this issue.

Basically, what happened is that I've had a 3rd party module that used gtts, which is a Google TTS package, (that internally uses requests) that was being loaded before my module using the steam package was, resulting in the monkey patch warning and generally the bot being unusable. The module loading order is non-deterministic and thus when the order was reversed back when I implemented the code above, everything worked fine. There's generally 4 options by which this issue can be fixed from my side:

• Get rid of gevent using modules, like this steam library - can't without sacrificing current bot's functionality
• Modify the bot's core code to apply monkey-patching before 3rd party modules are loaded - possible, but it voids all support from the bot maintenance team and requires me to reapply the patch every time the bot updates
• Ask every single 3rd party module creator using requests (or any other library that could use it) to add gevent patching code to their module - this would be needed since the module loading order is not deterministic, but it's not really feasible
• Avoid using anything else that uses unpatched requests entirely, and rely on the steam library doing the patching in the correct/expected order (the current approach I'm trying to go with)

Removing the 3rd party module that imported gtts has fixed the issue on my side, but as you can see, this whole monkey-patching requirement is really limiting. I should've been already avoiding anything that uses requests as it's blocking for asyncio code, and this gtts module was actually something I've only tested a couple of months ago and forgot about, hence why I had no real issue removing it. Still, my point stands - it'd be cool if this steam package would migrate to something less breaking and more popular like asyncio itself, if async code is needed here.

[rossengeorgiev]

There's two options here, either monkey patch before any import, or you could install steam from the master branch. The SteamClient no longer monkey patches by default, it is left to the user. I'll make a release soon with these unreleased changes

[DevilXD]

monkey patch before any import

As I said, I can't do that. This would require me to modify core bot's code, which doing so will void any support I can get from the maintenance team. The bot is an "install and use" type of a deal, and the only way to modify it's behavior is through modules, loading of which - as I said - is not deterministic, and the bot is already running when it tries to load everything.

install steam from the master branch

Wouldn't this still require patching for the client to work properly? Or is the patching completely optional then? I don't know gevent well enough to tell, I'm fine with blocking sockets as I'll be running this thing through an asyncio executor in a separate thread anyway, and the code is short enough to not make it an issue, hopefully.

[rossengeorgiev]

The gevent monkey patch simply makes the standard library cooperative. That means any waiting on IO, allows other gevent tasks to run. Without it, IO will block. SteamClient uses gevent primitives directly, so it remains cooperative. However, you will ocassionally need to yield to the gevent event loop in order for task to run. gevent.idle() or gevent.sleep() or any gevent based io. In your example, you are using the CDNClient, which uses requests, without monkey patching those requests will synchronous. Should be fine for your case.

I'm been thinking about how to decouple CDNClient from SteamClient. That could potentially help in cases like this, and also in potentially implementing async native version of SteamClient.

Eitherway, without monkey patching by default, there should be no problem using SteamClient is any codebase. As long as care is take to give chances for the gevent loop to run. Like an async task on 1s interval calling gevent.idle()