qasync memory leak

[syamajala]

I have an example with pyzmq that is leaking around ~60mb/sec of memory. When I run pyzmq with the standard asyncio event loop everything seems to work fine, but using a QEventLoop I see this problem.

import os
import asyncio
import time
import zmq
import zmq.asyncio
import numpy as np
from multiprocessing import Process
from collections import namedtuple
from PyQt5 import QtWidgets
from qasync import QEventLoop


Addr = namedtuple('Addrs', ['name', 'view'])
addr = Addr('graph', 'tcp://127.0.0.1:5557')


def run_worker():
    ctx = zmq.Context()
    socket = ctx.socket(zmq.PUB)
    socket.bind("tcp://127.0.0.1:5557")
    timestamp = 0

    while True:
        try:
            topic = 'view:graph:_auto_Projection.0.Out'
            socket.send_string(topic, zmq.SNDMORE)
            socket.send_pyobj(timestamp, zmq.SNDMORE)
            timestamp += 1
            msg = np.random.randn(1024, 1024)
            socket.send_pyobj(msg)

            time.sleep(0.1)
        except KeyboardInterrupt:
            break

    ctx.destroy()


async def update():
    ctx = zmq.asyncio.Context()
    sock = ctx.socket(zmq.SUB)
    sock.setsockopt_string(zmq.SUBSCRIBE, 'view:graph:_auto_Projection.0.Out')
    sock.connect(addr.view)

    while True:
        topic = await sock.recv_string()
        heartbeat = await sock.recv_pyobj()
        reply = await sock.recv_pyobj()
        print("PID:", os.getpid(), "RECEIVED:", reply)


if __name__ == "__main__":
    worker = Process(target=run_worker)
    worker.start()

    app = QtWidgets.QApplication([])
    loop = QEventLoop(app)
    asyncio.set_event_loop(loop)
    task = asyncio.ensure_future(update())

    try:
        loop.run_forever()
    finally:
        if not task.done():
            task.cancel()
        loop.close()

    # asyncio.run(update())

[syamajala]

Related to this it looks like there is a bug here: https://github.com/CabbageDevelopment/qasync/blob/master/qasync/init.py#L231

Some how we're ending up in a state where timerid has already been deleted from the __callbacks dictionary. should probably do if timerid in self.__callbacks: del self.__callbacks[timerid]

[syamajala]

I swapped out zmq for tcp and its not leaking anymore, so i think this proves the leak is coming from pyzmq.

[syamajala]

The pyzmq developers responded and this bug does appear to be due to qasync/asyncqt https://github.com/zeromq/pyzmq/issues/1406

[jordanbray]

Ok. I have tracked down the leak in this program (merge request incoming), but I figured I'd discuss all the primary issues here, while focusing on the fix itself in the MR.

The first issue (perhaps the primary issue), appears to be this bug: https://bugreports.qt.io/browse/QTBUG-92912

I believe the same issue exists within the startTimer function. This is being worked on upstream, but I, of course needed a solution now, so I updated this library.

The second issue is that, on Windows, this leak is actually causing the qasync library to run out of "User" objects, which QT creates internally for its timers. This can be verified by looking at the "Task Manager", going to the details page, right-clicking on the columns, and displaying the "User Objects" column. The value grows out of control, and will crash the application far before the system runs out of resources. In addition, the QT library prints out the following message to stdout:

QEventDispatcherWin32 the current process has used all of its system allowance of handles for Window Manager objects

I'm including that here for future googlers. I have written a patch for this, but frankly I think it needs to be reviewed by someone more knowledgeable.

[emdee-is]

Did a fix for this get merged? If so could you post the PR?

If not, could you post the patch?

[hosaka]

@jordanbray feel free to either PR your fix or post it here so I can look into integrating it until the upstream issue is resolved