Challenges with the STA server message loop and firing events

[bennyrowland]

I use comtypes for making several local servers to provide plugins for another application. I have a lot of challenges working with the MessageLoop class (https://github.com/enthought/comtypes/blob/76c35421c3d8f41e97db861209b4f640b0061b0b/comtypes/messageloop.py#L18) for single threaded apartments, specifically for firing events.

The MessageLoop basically blocks on GetMessage() all the time, until a message comes in, which then gets handled, and we go back to blocking on GetMessage() again. This means that it is basically impossible to do anything on the main thread, except inside the handler for a COM method. My server design requires the client calling methods which initiate long running calculations on the server but return immediately to avoid the client blocking on the method call.

It is possible to partially relieve this problem by adding a second thread in the server - the main thread gets the method from COM, puts it in a queue for the second thread to pick up when ready, then returns immediately. The issue then comes when the second thread wants to signal something back to the client via an event. Because of the STA model, only the main thread is able to call Fire_Event() to deliver the event to the client, but because that is blocked by the call to GetMessage(), we cannot actually fire the event (unless we wait for an incoming method call from the client and use that to fire any pending events).

Running in MTA mode allows firing the event from the second thread but this has other consequences for thread safety and I would rather avoid it if possible. Therefore I am looking for suggestions on how we might make firing events from servers a more straightforward task. At the moment my solution is to have an intermediate "event-firing" server, which provides an interface of methods which can be called which immediately synchronously fire the corresponding event. An instance of this server is created by the client (which connects to its events) and passed to the primary server, which marshals it over to the secondary thread (via the GIT) which is then able to call its methods and fire events on the client. This works, but is obviously a sub-optimal design.

What I (currently) think would be nicest would be some way of getting in to the MessageLoop internally without a COM method call coming in. I think it would be really cool if we could interface the loop with cooperative multitasking like asyncio, then it would be easy to run other tasks on the main thread interspersed with handling COM method calls. A secondary option might be to decouple Fire_Event so that it can be called in a thread-safe manner - when a secondary thread calls Fire_Event it could put the event in a queue, then post a message in the message loop that would trigger a handler on the main thread to actually fire queued events.

I would also be very happy to hear any other suggestions or alternative approaches to solving this problem. Thanks for your input.