Continuous memory consumption with large scripts

[SteveUM]

IP 2.7.7

When embedding IPy into C# application, large scripts that include many list comprehensions continue to use memory even after the script has finished executing. This leads eventually to OOM exception. When this occurs, the following thread continues to execute and consume memory:

Not Flagged 0x00003064 0x0000001E Worker Thread Worker Thread System.Core.dll!System.Linq.Expressions.Compiler.LambdaCompiler.CreateDelegate Normal

if you freeze the thread, memory consumption stops.

What is really strange is that if you insert a sys.exit prior to the list comprehensions you get the same OOM result as if you executed the list comprehensions.

Executing scriptsource or compiled gives same result.

Using the same script with smaller number of list comprehensions will continue to use memory, but then releases memory after a few seconds.

I am unable to provide my script as it's intertwined with many of our C# support objects, but the script structure is as follows.

# ~1800 lines of python code
...

# optional sys.exit()

# ~400 list comprehensions
l = [int(x) for x in range(0, 100)]
...
l = [int(x) for x in range(0, 100)]

If anyone knows of a way to work around this, I would appreciate any suggestions.

[slide]

There has been some significant work that will be part of 2.7.8, is it possible for you to grab the alpha release a d try it out to see if it fixes your issue?

[SteveUM]

I will test this issue against the 2.7.8 beta and report findings.

[SteveUM]

Finally got a chance to test this against 2.7.8 binaries. Same problem. Found more information though. The problem starts in the LightThrowCompile method in FunctionCode.cs. When the callback function is called by threadpool, the LambdaExpression.Compile() function hangs and never returns, continuously using memory until an OOM exception. I don't really understand the LightThrowCompile function, but the first if statement seems a bit suspect:

        internal void LightThrowCompile(CodeContext/*!*/ context)
        {
            if (++_exceptionCount > 20)
            {...

What's the significance of _exceptionCount > 20? If I set the value to something larger than 20, my memory issue goes away. Anyone know why the function ignores the first 20 calls?

[SteveUM]

We've had more time in look into this issue. It appears that this problem occurs when:

• hosting IPy from a C# application
• a hosted python script calls a C# function and the function throws an exception
• the script is large (smaller looping structures don't create the problem)
• issue reproduced in 2.7.7, 2.7.8, 2.7.9 (Windows 7)

It's been difficult to come up with a test script to reproduce this issue because our scripts use many of our custom functions, etc. But, we can partially reproduce with the attached file. What happens with this script is that after the script execution ends, you can see memory consumption continue for a time, then top out. In our in-house scripts, this issue is much more severe due to the size and complexity of the scripts, where we eventually get OOM. The script is simple and looks like:

import sys
import clr
import time

clr.AddReference("HelperFunction.dll")
from HelperFunction import *

try:
    pyHelper.logError("Testing..")
except Exception as ex:
    print str(ex)
try:
    pyHelper.logError("Testing..")
except Exception as ex:
    print str(ex)
# copy the above for about 4-5K lines...

To demonstrate, the script uses a C# helper function that simply throws an exception

    public static class pyHelper
    {
        public static void logError(string error)
        {
            throw new Exception("A problem occurred while logging error message.");
        }
    }

We are investigating a way to turn off the light throw compile by creating a new option NoCompileLightEx. Besides adding it to the command line parsing code, it only modifies the following in FunctionCode.cs.

internal void LightThrowCompile(CodeContext/*!*/ context) {
    //if (++_exceptionCount > 20) { //original line
    if (++_exceptionCount > 20 && !context.LanguageContext.PythonOptions.NoCompileLightEx) {
        if (!_compilingLight && (object)Target == (object)LightThrowTarget) {
    ...

Our initial testing with this is good, where the behavior of the script appears to be valid and the memory issue is resolved. We will continue to test with this change and follow up.