Ctypes ReadOnlyMemoryError

[MichaelRThompson]

I am currently running into an issue using pyjulia. I am performing a nonlinear optimization of an objective function that requires some heavy numerical integration within the objective function. When it was in pure Python, I had it working just fine, but it was understandably slow, and I have switched the numerical integration to Julia.

The overall code is too long to reproduce here, but the general workflow is:

• nlopt called from Python
• Within the objective function, numerical integration is performed in Julia and returned to Python
• Within the objective function, Scipy's fsolve() is used to solve a nonlinear system of equations based on the output
• Basic math is performed on the output from fsolve, and returned to nlopt

Without any attempts to debug, this optimizer will iterate for approximately 15 minutes (a couple thousand iterations) before I get a ReadOnlyMemoryError. I've pasted the output at the bottom of this issue.

I've tried a good number of ways of overcoming this error, but only seem to be able to delay it for a time:

• Switching the optimization algorithm within nlopt. No change.
• Importing julia within the "master" python script before any other modules. No change.
• Putting a time.sleep() before and after the julia numerical integration. My thought being that julia may be too fast for python in an iteration like this, and it may be going too fast for the Global Interpreter Lock to be released. It usually takes longer for the error to appear, but it still inevitably shows up, usually after around 30 minutes.
• Performing regular Julia garbage collection Base.GC.gc() at some cadence within the iteration. Generally speaking, the more rapid the cadence, the longer it will last before running into the error, but the slower the iteration. Performing garbage collection after every objective function evaluation is the only way I've found to make the optimization last an entire hour (which is what I currently have as the timeout time for the optimizer).
• Combinations of the previous two steps. Time.sleep() and Julia garbage collection. Currently by best attempt is performing garbage collection every 100 iterations and a pause of 0.001 seconds every iteration.

The program isn't using significant RAM, so I'm doubtful of a out of memory issue.

Versions of relevant modules: Python 3.7.2 Pyjulia 0.4.1 numpy 1.16.4 scipy 1.3.0 nlopt 2.6.1 Windows

Error message:

fatal: error thrown and no exception handler available.
ReadOnlyMemoryError()
DllCanUnloadNow at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\DLLs\_ctypes.pyd (unknown line)
unknown function (ip: 00007FFD716C6D79)
PyObject_FastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyFunction_FastCallDict at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallDict at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyObject_FastCallDict at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyObject_CallFunctionObjArgs at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyNumber_Invert at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyArg_UnpackStack at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyObject_GetAttr at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyFunction_FastCallDict at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyObject_Call at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyInit__minpack at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\_minpack.cp37-win_amd64.pyd (unknown line)
PyInit__minpack at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\_minpack.cp37-win_amd64.pyd (unknown line)
fdjac1 at C:\projects\scipy-wheels\scipy\scipy/optimize/minpack\fdjac1.f:112
hybrd at C:\projects\scipy-wheels\scipy\scipy/optimize/minpack\hybrd.f:226
PyInit__minpack at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\_minpack.cp37-win_amd64.pyd (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyFunction_FastCallDict at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyObject_SetAttr at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyFunction_FastCallDict at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyObject_Call at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
unknown function (ip: 00007FFD5CAD2A83)
unknown function (ip: 00007FFD5CAD335B)
unknown function (ip: 00007FFD5B086069)
unknown function (ip: 00007FFD5B085DD9)
nlopt_optimize at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\lib\site-packages\nlopt\nlopt.dll (unknown line)
nlopt_optimize at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\lib\site-packages\nlopt\nlopt.dll (unknown line)
unknown function (ip: 00007FFD5CAD36ED)
PyInit__nlopt at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\lib\site-packages\nlopt\_nlopt.pyd (unknown line)
unknown function (ip: 00007FFD5CAC78C2)
PyErr_NoMemory at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyObject_SetAttr at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCodeEx at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyEval_EvalCode at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyFuture_FromASTObject at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyRun_FileExFlags at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyRun_SimpleFileExFlags at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyRun_AnyFileExFlags at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
Py_UnixMain at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
Py_UnixMain at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
PyErr_NoMemory at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
Py_Main at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
Py_Main at C:\Users\mthompson\AppData\Local\Programs\Python\Python37\python37.dll (unknown line)
unknown function (ip: 00007FF716F61257)
BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)

[tkf]

I think it's likely a problem in PyCall (e.g., it's GC interop). You can skip PyJulia part quite easily by running something like

# Define your Julia function
function julia_function(....)
    ...
end

using PyCall
py"""
from your_module import your_function
your_function(julia_function=$julia_function)  # pass `julia_function`
"""

in a Julia script.

To debug the actual problem, it's useful to know what kind of objects are passed to/returned from your Julia function. Are they just Numpy arrays or scalars (if so, this problem would be much harder to solve...)? Does it include some custom Python class or Julia structs (if so, avoiding them could help)?

[MichaelRThompson]

The Julia function is essentially a wrapper around DifferentialEquations.jl that sets up my own set of equations. So what's being passed to the function is:

• A numpy array of 6 initial conditions
• A tuple of the time span (start_time, end_time)
• A string to specify which set of equations to use (there are a couple possibilities in library, although I'm only using one of them for this application)
• A tuple of additional numerical parameters

It's returning a list of lists of numerical values. A length 6 list (for the 6 states) where each element is the time history of that state.

[MichaelRThompson]

Update, I moved to a different machine (still Windows) and did a fresh install of Python, Julia, etc.

Ran into the same issue after around 25 minutes of iteration. It made it to a higher number of iterations than I've ever seen, but that could be because the machine is nicer than the one I typically use.

[MichaelRThompson]

Something that I've currently tried (with no success) is a try/except statement that performs numerical integration in julia unless there is an error, and then switches over to python if there is.

Unfortunately, the error doesn't seem to be caught by this, and the optimization still crashes.

Do you know if pyjulia is set up to return exceptions to python at all?

[tkf]

I see. They seem to be safe objects to pass around.

How do you get the Julia function in the objective function? Do you do something like obj.julia_function(*some_arguments) in Python? If so, avoiding the attribute access could help. See also https://github.com/JuliaPy/pyjulia/issues/306#issuecomment-505634270

Other than that, I don't know what's the best way to debug ReadOnlyMemoryError. Maybe open an issue in Julia's bug tracker? Ping me from there if you do.

[MichaelRThompson]

Yes, I have been loading in a custom Julia module and then calling one script within it like that.

I took the advice given in your linked comment and re-coded the call to differential equations inside a Main.Eval() call and had a very unexpected result: Not only have I not encountered the issue since, the code is also much faster. Ten times faster for one particular test case. Not quite as fast for others, but definitely still a performance increase.

I've ran multiple optimization algorithms through hundreds of thousands of iterations and multiple hours without coming across the issue again.

There must be some type of weird overhead by using the previous syntax.

Do you think that reporting the ReadOnlyMemoryError would be better suited for Julia or PyCall?

[tkf]

Not only have I not encountered the issue since, the code is also much faster.

Ah, that's great!

There must be some type of weird overhead by using the previous syntax.

In general (not just PyJulia), there is some cost whenever you write obj.attr in Python so caching it to a local variable can make things faster in principle; usually not much. However, this cost is much bigger in PyJulia because it would create a custom object that is tracked in both Julia's and Python's GC. It also invokes some dynamic type conversion in Julia's side which takes some time as well.

Do you think that reporting the ReadOnlyMemoryError would be better suited for Julia or PyCall?

I think @stevengj is notified by this issue so I guess there is not much point to make an issue in PyCall. But if you can create a minimal working example to invoke ReadOnlyMemoryError without using PyJulia (using, e.g., py"...." as I suggested above), please open a new issue in PyCall and close this one.

[MichaelRThompson]

I'll see if I can reproduce it in the coming days with a simple example. Thanks for the help.

[dpinol]

@tkf thanks for proposing the cache workaround. In my case it also runs much faster and I haven't had more crashes. For the record, I include my shim to cache all members of the Julia module

from inspect import getmembers

from julia.core import JuliaModule

class JuliaShim:
    def __init__(self, module: JuliaModule):
        for name, attr in getmembers(module):
            self.__setattr__(name, attr)