Intermittent crashes on windows with python 3.9

[bsteffensmeier]

Describe the bug In AppVeyor we are seeing intermittent crashes in the python3.9 build of the dev_4.0 branch while running the test in TestCrossLangSync.java. The output from that test is below.

To Reproduce Simply run setup.py test on windows with python 3.9.9. It only fails about 50% of the time so repeating 2 or 3 times may be necessary.

Expected behavior All tests should pass on python 3.9

Environment (please complete the following information):

• OS Platform, Distribution, and Version: Windows
• Python Distribution and Version: 3.9.9
• Java Distribution and Version: java 11 or java 17
• Jep Version: dev_4.0
• Python packages used (e.g. numpy, pandas, tensorflow): none

Additional context Output on failure:

Traceback (most recent call last):
File "C:\projects\jep\src\test\python\test_synchronized.py", line 16, in test_synchronized
jep_pipe(build_java_process_cmd('jep.test.synchronization.TestCrossLangSync'))
File "C:\Python39-x64\lib\contextlib.py", line 263, in helper
return _GeneratorContextManager(func, args, kwds)
File "C:\Python39-x64\lib\contextlib.py", line 87, in __init__
self.gen = func(*args, **kwds)
File "C:\projects\jep\src\test\python\jep_pipe.py", line 38, in jep_pipe
assert False, stdout
AssertionError: b'#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ff851c9f83f, pid=3832, tid=4928
#
# JRE version: OpenJDK Runtime Environment (11.0.2+9) (build 11.0.2+9)
# Java VM: OpenJDK 64-Bit Server VM (11.0.2+9, mixed mode, tiered, compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# C  [python39.dll+0x9f83f]
#
# Core dump will be written. Default location: C:\\projects\\jep\\hs_err_pid3832.mdmp
#
# An error report file with more information is saved as:
# C:\\projects\\jep\\hs_err_pid3832.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#'

[ndjensen]

I was able to repeat this on Windows with Python 3.9.8 and 3.9.9, but not with 3.9.7. With that knowledge, I started building Python from source on Windows and testing different commits, tracking which commits experienced the crash and which did not. I have determined that this commit https://github.com/python/cpython/commit/52d9d3b75441ae6038fadead89eac5eecdd34501 introduced the change that is causing these crashes. Most of that looks like test code except the change to the if statement in import.c. All that said, I don't understand the code well enough to understand why that can cause a crash, why it doesn't happen every time, and why it only happens on Windows. But the Python built on the commit before that one does not experience the crash, and Python built on that commit does.

For reference I also tested this in a Linux docker container and was unable to repeat it, so I agree, this only happens on Windows.

[bsteffensmeier]

I think this is caused by a python bug: https://bugs.python.org/issue46070

I have been able to reproduce the issue without any native/jep code using the following python. This python crashes about 50% of the time on windows on python 3.9.9(and likely 3.9.8 although I have not tested.)

import _testcapi
import threading

code = """
import importlib.util
"""

def doIt():
    _testcapi.run_in_subinterp(code)

tt=[]

for i in range(16):
    t = threading.Thread(target=doIt)
    t.start()
    tt.append(t)

for t in tt:
    t.join()