IronPython should have well defined entry and exit points for Python code and should clear exceptions at this boundaries (memory leak w/ unhandled exceptions)

[ironpythonbot]

Currently IronPython leaks memory when exceptions go repeatedly unhandled as they exit Python code. This is because we are constantly growing a list of exception frames and it only ever gets cleared when the exception is caught by Python code.

Instead we should frame all Python code so that we can properly associate the exception data when we leave a chain of Python code. This means that all external entry points to Python code need to be well defined. This includes when we convert functions and classes to delegates and top-level script code. We will also need to trap calls to Python functions and classes from foreign languages.

In debug builds all Python code should assert that it is properly setup for exception handling. This way we can make sure that there are no missing leaks.

Once this in place it could also form the infrastructure for moving off of .NET exceptions within pure Python code so that we can have fast exception support (at a cost of a small amount of runtime throughput).

using System;  
using System.Threading;  
using IronPython.Hosting;  
using IronPython.Runtime;  
using IronPython.Compiler;  
using System.Collections.Generic;  
using Microsoft.Scripting.Hosting;  

namespace IPyTest  
{    
    class Program
    {
        static void Main(string[] args)
        {
            bool cont = true;
            while (cont)
            {
                var ipy = new IPy();
                try
                {
                    // Set the below boolean to "false" to run without a memory leak
                    // Set it to "true" to run with a memory leak.
                    ipy.run(true);
                }
                catch { }
            }
        }
    }

    class IPy
    {
        private string scriptWithoutLeak = "import random; random.randint(1,10)";
        private string scriptWithLeak = "raise Exception(), 'error'";

        public IPy()
        {
        }

        public void run(bool withLeak)
        {
            //set up script environment
            Dictionary<String, Object> options = new Dictionary<string, object>();
            options["LightweightScopes"] = true;
            ScriptEngine engine = Python.CreateEngine(options);
            PythonCompilerOptions pco = (PythonCompilerOptions) engine.GetCompilerOptions();
            pco.Module &= ~ModuleOptions.Optimized;
            engine.SetSearchPaths(new string[]{
                @"C:\Program Files\IronPython 2.6\Lib"
            });
            ScriptRuntime runtime = engine.Runtime;
            ScriptScope scope = runtime.CreateScope();
            var source = engine.CreateScriptSourceFromString(
                withLeak ? scriptWithLeak : scriptWithoutLeak
            );
            var comped = source.Compile();
            comped.Execute(scope);
            runtime.Shutdown();
        }
    }
}  

Work Item Details

Original CodePlex Issue: Issue 25478 Status: Active Reason Closed: Unassigned Assigned to: Unassigned Reported on: Nov 30, 2009 at 7:26 PM Reported by: dinov Updated on: Oct 9 at 12:25 PM Updated by: hfoffani Thanks: Jonathan Howard

[ironpythonbot]

On 2010-03-04 22:03:06 UTC, JErasmus commented:

This requires urgent attention as the amount of memory leaked highly depends on what occurs within the script.

For example: Running the following script will easily leak over 10Mb in 135 iterations (one minute on my PC), and continues leaking if run for longer.

from System.Collections.Generics import List list = Listint list.AddRange(range(1, 1000000)) raise Exception(), 'error'

The issue can be temporarily fixed by clearing the dynamic stackframes after catching the exception in the host program:

try { comped.Execute(scope); } catch { IronPython.Runtime.Operations.PythonOps.ClearDynamicStackFrames() }

With this fix, the 135 iteration test only resulted in a maximum of 565Kb memory usage.

[hfoffani]

EDIT: Fixed the totally misleading previous version of my commentary.

I was NOT able to reproduced it with IronPython 2.7.6rc2 under Windows 10 Professional using .NET Framework 4.6.2

I couldn't reproduce the reported leak. The consumption for a 2000 iterations loop is between 8 and 16 MB with no difference when raising the exception.

There could be another leak though as I couldn't reproduce the 600KB usage reported by JErasmus.

This test uses the csharp from Dino and the IPy code from JErasmus

Source:

----- cut test776.cs -------

using System;
using System.Threading;
using IronPython.Hosting;
using IronPython.Runtime;
using IronPython.Compiler;
using System.Collections.Generic;
using Microsoft.Scripting.Hosting;

namespace IPyTest
{

class Program
{
    static void Main(string[] args)
    {
        bool cont = true;
        while (cont)
        {
            var ipy = new IPy();
            try
            {
                // Set the below boolean to "false" to run without a memory leak
                // Set it to "true" to run with a memory leak.
                ipy.run(true);
            }
            catch { }
        }
    }
}


class IPy
{
    private string scriptWithoutLeak = "import random; random.randint(1,10)";
    private string scriptWithLeak = @"
from System.Collections.Generics import List 
list = Listint 
list.AddRange(range(1, 1000000)) 
raise Exception(), 'error'
";

    public IPy()
    {
    }

    public void run(bool withLeak)
    {
        //set up script environment
        Dictionary<String, Object> options = new Dictionary<string, object>();
        options["LightweightScopes"] = true;
        ScriptEngine engine = Python.CreateEngine(options);
        PythonCompilerOptions pco = (PythonCompilerOptions) engine.GetCompilerOptions();
        pco.Module &= ~ModuleOptions.Optimized;
        engine.SetSearchPaths(new string[]{
            @"C:\Program Files\IronPython 2.7\Lib"
        });
        ScriptRuntime runtime = engine.Runtime;
        ScriptScope scope = runtime.CreateScope();
        var source = engine.CreateScriptSourceFromString(
            withLeak ? scriptWithLeak : scriptWithoutLeak
        );
        var comped = source.Compile();
        comped.Execute(scope);
        runtime.Shutdown();
    }
}
}
---------------------------

Build and test with:

------ cut test776.bat ----
C:\Windows\Microsoft.NET\Framework\v4.0.30319\csc.exe test776.cs /reference:"c:\Program Files (x86)\IronPython 2.7\IronPython.dll"  /reference:"c:\Program Files (x86)\IronPython 2.7\Microsoft.Scripting.dll"
test776.exe
-----------------------

by @hfoffani

[hfoffani]

The proposed workaround doesn't work because IronPython.Runtime.Operations.PythonOps.ClearDynamicStackFrames() is not available now.

by @hfoffani

[hfoffani]

Same behavior in 2.7.6.3 under windows 8.1 As gist: https://gist.github.com/hfoffani/509d7ceaf3d49d143bec1695979ae345