Flask + ProcessPoolExecutor -> TypeError: can't pickle _thread.lock objects

[codeape2]

• Python version: 3.7.7 (default, Apr 15 2020, 05:09:04) [MSC v.1916 64 bit (AMD64)]
• aiohttp: 3.6.2
• aiohttp_wsgi: 0.8.2
• flask: 1.1.2

To reproduce:

from concurrent.futures.process import ProcessPoolExecutor
from concurrent.futures.thread import ThreadPoolExecutor

from aiohttp import web
from aiohttp_wsgi import WSGIHandler
from flask import Flask


flaskapp = Flask(__name__)


@flaskapp.route('/')
def index():
    return 'hello from flask'


executor = ProcessPoolExecutor()  # ThreadPoolExecutor() works


aioapp = web.Application()
aioapp.router.add_route('*', '/{path_info:.*}', WSGIHandler(flaskapp, executor=executor))


if __name__ == '__main__':
    web.run_app(aioapp)

Traceback:

======== Running on http://0.0.0.0:8080 ========
(Press CTRL+C to quit)
Error handling request
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "F:\envs\_april\lib\multiprocessing\queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "F:\envs\_april\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "F:\envs\_april\lib\site-packages\aiohttp\web_protocol.py", line 418, in start
    resp = await task
  File "F:\envs\_april\lib\site-packages\aiohttp\web_app.py", line 458, in _handle
    resp = await handler(request)
  File "F:\envs\_april\lib\site-packages\aiohttp\web_urldispatcher.py", line 158, in handler_wrapper
    return await result
  File "F:\envs\_april\lib\site-packages\aiohttp_wsgi\wsgi.py", line 266, in handle_request
    environ,
  File "F:\envs\_april\lib\multiprocessing\queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "F:\envs\_april\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects

[codeape2]

A minimal wsgi app also fails:

def simplest_wsgi_app(environ, start_response):
    print(environ)
    start_response('200 OK', [('Content-Type', 'text/plain')])
    return [b"hello from wsgi!\n"]

With TypeError: can't pickle _asyncio.Future objects

[codeape2]

The minimal wsgi app works if I use a modified WSGIHandler:

class MultiprocessCompatibleWSGIHandler(WSGIHandler):
    def _get_environ(self, request, body, content_length):
        environ = super()._get_environ(request, body, content_length)
        del environ['asyncio.loop']
        del environ['asyncio.executor']
        del environ['aiohttp.request']
        return environ

... but flask does not work. I guess serializing the entire flask application is not feasible. A possible approach for using a ProcessPool executor is to initialize the wsgi app in the process pool's initializer.