Change back the force shutdown of the executor

[basnijholt]

Change back to self.executor.shutdown(wait=True) here when https://bugs.python.org/issue36281 is fixed.

Related https://github.com/python-adaptive/adaptive/issues/156.

[eric-burel]

Hi, did you ever solved this one? I am on Python project with the exact same issue, do you know if an update of Python can fix this?

[akhmerov]

The issue doesn't occur for me on Python 3.8.3 or 3.9.5; what is your Python version?

[eric-burel]

Thanks! I am running 3.6.9 and I try updating to 3.7.0, I update carefully and step by step because it's a big and old project, so I try to find the minimal path to fix this issue.

[akhmerov]

~~3.7.0 is unlikely going to help, according to the issue description.~~

[akhmerov]

Disregard my last message, I misread. The workaround is implemented on Python version above 3.7, however it should not impact 3.6. Also note that you can avoid it entirely by calling a runner with shutdown_executor=False (the default).

Are you sure that you are seeing this issue actually?

[eric-burel]

I think so, I have the same message and same diagnosis (it worked with 3.6.9 and fails with 3.7.0). I have event that are not processed because of the closed handle. But I haven't digged that much, I remember suffering to produce this code and it was around 4 years ago haha. I'll try just updating to > 3.7.0 anyway

[eric-burel]

For the record, I seem to reproduce in Python 3.8, however I am not using an executor, code looks like this:

        start_server = websockets.serve(send_data_to_client, host, port)
        self.socket_server = start_server

        # need to pass the loop because the socket server lives in another thread
        loop = asyncio.get_event_loop()

        # start the server
        def start_socket_server():
            loop.run_until_complete(start_server)
            loop.run_forever()

        self.socket_thread = Thread(
            target=start_socket_server, daemon=True, name="socket_server")
        logger.info("Starting socket server on {}:{}".format(host, port))
        self.socket_thread.start()

No idea yet how to adapt your fix, I guess I need to close the thread or smth.

[basnijholt]

The "fix" is just doing:

runner = adaptive.Runner(learner, goal=lambda l: l.loss() < 0.01, shutdown_executor=True)

or

runner = adaptive.Runner(learner, goal=lambda l: l.loss() < 0.01, shutdown_executor=False)

I don't really understand what the problem is perhaps?

[eric-burel]

I am not using directly this lib actually sorry, but just hit the same downstream issue, that's why I am posting here because you seem to be the most active project regarding this. But I think I progress: the issue is not in the way I run my event consumption loop, but the way I setup multiprocessing in the app.