Add support for Python awaitables.

[Hoikas]

trafficstars

This changeset allows the Python scripts to use coroutines and await responses from engine API calls from PEP 492. The benefit to this is that some of the Python callback soup can be eleminated. To illustrate this, I removed a callback from xTelescope to make the logic clearer:

diff --git a/Scripts/Python/xTelescope.py b/Scripts/Python/xTelescope.py
index 380049e8fc..1c69ae23d4 100644
--- a/Scripts/Python/xTelescope.py
+++ b/Scripts/Python/xTelescope.py
@@ -211,15 +212,13 @@ def IQuitTelescope(self):
         boolScopeOperator = 0
         self.SDL["boolOperated"] = (0,)
         self.SDL["OperatorID"] = (-1,)
-        #Re-enable first person camera
-        cam = ptCamera()
-        cam.enableFirstPersonOverride()
-        PtAtTimeCallback(self.key,3,1) # wait for player to finish exit one-shot, then reenable clickable
-        PtDebugPrint("xTelescope.IQuitTelescope:\tdelaying clickable reenable",level=kDebugDumpLevel)
-        
-    def OnTimer(self,id):
-        if id==1:
-            Activate.enable()
-            PtDebugPrint("xTelescope.OnTimer:\tclickable reenabled",level=kDebugDumpLevel)
-            PtSendKIMessage(kEnableKIandBB,0)
 
+        # Re-enable first person camera
+        virtCam.enableFirstPersonOverride()
+
+        # wait for player to finish exit one-shot, then reenable clickable
+        PtDebugPrint("xTelescope.IQuitTelescope:\tdelaying clickable reenable", level=kWarningLevel)
+        await PtSleep(3.0)
+        Activate.enable()
+        PtSendKIMessage(kEnableKIandBB, 0)
+        PtDebugPrint("xTelescope.OnTimer:\tclickable reenabled", level=kWarningLevel)

The introduced API is somewhat analagous to Python's asyncio module. This module is inappropriate for use in Plasma because it implements its own event loop that controls an entire thread. In Plasma, the engine's main loop is the event loop. Any pending coroutines (async def in scripts that have not completed) are pumped when plEvalMsg is sent. I have implemented the following awaitable functions:

• PtSleep(): delays execution of a coroutine for a requested amount of time. This is useful for replacing a lot of delay timers.
• PtWaitFor(): Analagous to asyncio.wait(), this waits for all supplied awaitables to finish (with an optional timeout)... or it returns on the first error.

Both of these functions accept an optional keyword argument block=True that turns them into synchronous, main thread blocking operations.

This functionality is implemented using a new type ptAsyncTask that implements the awaitable protocol. By default, a ptAsyncTask is analogous to an asyncio.Future that can have a result set/fetched and can be awaited. The Python glue code can optionally switch in different task backends that run arbitrary C++ code every engine eval. Like PtSleep() and PtWaitFor(), ptAsyncTask.getResult() also accepts an options keyword argument block=True to allow blocking the main thread on a result.

This changeset is largely the bare minimum to implement this design. The ultimate goals is to allow cleaner code when performing tasks from Python that have known delayed responses, especially when contacting the server. There are many places that still block the render loop that can be improved by returning a ptAsyncTask and awaiting the result. I would also like to use this in new code for restoring support for MOULa-style marker games.

There are currently two items that may require additional investigation:

• Pending script coroutines increment the object reference counter of the plPythonFileMod. This means that when an awaitable completes, it may be executing in a context where the script is executing in a "key-only" context, meaning there are no PRP objects loaded, and it will be shouting messages into the void. This should be fine, but we'll want to carefully make sure nothing in pfPython assumes non-global objects are loaded. Making it work this way is a lot simpler than adding support for cancellation.
• The ptAsyncTask type hint in Plasma.py doesn't have a way to specify the result type of the awaitable. Because of that, I've simply been using typing.Awaitable for the hints for functions that return ptAsyncTask.

[dgelessus]

This module is inappropriate for use in Plasma because it implements its own event loop that controls an entire thread. In Plasma, the engine's main loop is the event loop.

FWIW, it is possible to write a custom asyncio event loop implementation that integrates with a non-native event loop. See here for example. Unfortunately I've only just started learning asyncio and don't know anything about Plasma's event loop, so I can't say if this would be actually possible here.

• The ptAsyncTask type hint in Plasma.py doesn't have a way to specify the result type of the awaitable. Because of that, I've simply been using typing.Awaitable for the hints for functions that return ptAsyncTask.

I think you can simply make ptAsyncTask extend Awaitable and it will inherit the type parameter:

class ptAsyncTask(Awaitable):
    # ...

Or possibly you might have to explicitly re-define the type parameter (I don't remember exactly):

R = typing.TypeVar("R")
class ptAsyncTask(Generic[R], Awaitable[R]):
    # ...

Or if neither of that works for some reason, you can hint the __await__ method manually:

R = typing.TypeVar("R")
class ptAsyncTask(typing.Generic[R]):
    def __await__(self) -> Generator[Any, None, R]:
        ...

[Hoikas]

Thanks for the info about type hints. I'll play around with that soon and see what I can come up with.

FWIW, it is possible to write a custom asyncio event loop implementation that integrates with a non-native event loop. See here for example. Unfortunately I've only just started learning asyncio and don't know anything about Plasma's event loop, so I can't say if this would be actually possible here.

I'm still not sure I like the idea of dragging in all of the networking-geared stuff from asyncio. We really don't need all of that complexity.

[Hoikas]

The type hints should now be correct, and I've fixed a few other problems, one of which was a crashing bug.

[Hoikas]

The carnage has spread to PtYesNoDialog. To understand how this is beneficial, see the changes to nxusBookMachine.py.

[Hoikas]

I'm converting this to a draft because I want to split part of the second commit out into a new branch to use in the game manager rewrite. I do not intend to use the awaitable parts of this PR for the game manager anymore.

[Hoikas]

This now depends on (and is blocking on) #1179.

[Hoikas]

This is still a cool idea, but it's got lots of conflicts and maturing. Closing.