Concurrent complete and execute_interactive hang forever

[AyWa]

Info / version

python: 3.6.5 jupyter / ipython related:

ipykernel                          4.9.0
ipython                            5.8.0
jupyter                            1.0.0
jupyter-client                     5.2.3
jupyter-console                    5.2.0
jupyter-core                       4.4.0

Bug

It seems like execute_interactive can hang forever if an complete call is made at the same time. Let me know if I am wrong but this is what I understand:

• execute_interactive and complete are safe to call in different thread (but on the same kernel client) (as far as I know it seems right but let me know if I am wrong and need to have logic to protect that)
• complete and execute_interactive will never really run at the same time (in the sense of complete will wait for execute to complete before run)

So I think this is a bug, but let me know if I am using badly the client. (and then maybe we can update the doc or put a warnings somewhere), if the bug is verify then we can work on a fix :)

Code to reproduce

setup

importing the library and define 2 thread: MyThread will just execute the code and print result when execution finish MyThread2 will just do one autocomplete and wait for result (reply=true)

import jupyter_client
import threading
import time

code = "import time\n" + "print(1)\n" + "time.sleep(10)\n" + "print(2)"
autoCode = "time."

def MyThread ():
    print("will start execute")
    reply = kc.execute_interactive(code)
    print("reply execute", reply)

def MyThread2 ():
    time.sleep(2)
    print("will start auto complete")
    auto_reply =kc.complete(autoCode, len(autoCode), reply=True)
    print("reply auto complete", auto_reply)

Main code

then we start a new kernel start the 2 thread wait for everything to complete (note that we sleep in thread 2 to ensure that the thread 1 is already running)

km, kc = jupyter_client.manager.start_new_kernel(kernel_name='python')
t1 = threading.Thread(name="1", target=MyThread)
t1.start()
t2 = threading.Thread(name="2", target=MyThread2)
t2.start()
t1.join()
t2.join()
print("everything is done")

Current behavior

• execute_interactive will never return
• complete sometimes return sometimes not

sometimes:

will start execute
1
will start auto complete
2

or sometimes:

will start execute
1
will start auto complete
2
reply auto complete {'header': {'version': '5.3', 'date': datetime.datetime(2019, 3, 12, 1, 35, 37, 878053, tzinfo=tzutc()), 'session': 'b92ea2bc-a8683e7eb169da1ffa53588b', 'username': 'marc', 'msg_type': 'complete_reply', 'msg_id': '3eed1bd0-e21cd0a18fc77ce1a879b43f'}, 'msg_id': '3eed1bd0-e21cd0a18fc77ce1a879b43f', 'msg_type': 'complete_reply', 'parent_header': {'session': '50688e12-25da93f628e6687db2afb9ae', 'username': 'marc', 'msg_type': 'complete_request', 'msg_id': 'c2976630-2e8d964e1090fc8302550e8a', 'version': '5.3', 'date': datetime.datetime(2019, 3, 12, 1, 35, 33, 866406, tzinfo=tzutc())}, 'metadata': {}, 'content': {'matches': ['time.altzone', 'time.asctime', 'time.clock', 'time.clock_getres', 'time.clock_gettime', 'time.CLOCK_MONOTONIC', 'time.CLOCK_MONOTONIC_RAW', 'time.CLOCK_PROCESS_CPUTIME_ID', 'time.CLOCK_REALTIME', 'time.clock_settime', 'time.CLOCK_THREAD_CPUTIME_ID', 'time.ctime', 'time.daylight', 'time.get_clock_info', 'time.gmtime', 'time.localtime', 'time.mktime', 'time.monotonic', 'time.perf_counter', 'time.process_time', 'time.sleep', 'time.strftime', 'time.strptime', 'time.struct_time', 'time.time', 'time.timezone', 'time.tzname', 'time.tzset'], 'cursor_end': 5, 'cursor_start': 0, 'metadata': {}, 'status': 'ok'}, 'buffers': []}

Expected behavior

• execute_interactive return when finish to execute
• complete return when finish to execute so in term of log this is what I expect:

will start execute
1
will start auto complete
2
reply execute {...}
reply auto complete {...}
everything is done