Escape Analysis can incorrectly think an object does not escape during recursive calls

[IBMJimmyk]

Escape Analysis tries to peek into methods to check to see if objects escape inside of those methods. For recursive methods, it only checks 1 recursion deep before setting ignoreRecursion. It incorrectly assumes that if an object did not escape during one layer of recursion that it will not escape at all:

https://github.com/eclipse-openj9/openj9/blob/332a5d30b53ea0e197470b6a9b90ca0bb0502b12/runtime/compiler/optimizer/EscapeAnalysis.cpp#L5107-L5114

I created a test case where different objects escape during each layer of recursion. The key to focus on is rotateEscape. The first time it is called, the original rt0 escapes. The second time, rt1. Then rt2 and rt3.

Since EA only checks two layers deep it never figures out that rt2 escapes if rotateEscape recurses exactly 3 times.

Test case:

public class RecursionTest {

    public static RecursionTest globalRT = null;

    public int initValue = 0;

    public static void main (String[] args) {

        RecursionTest t = new RecursionTest(99);

        for (int i = 0; i < 100; i++) {
            t.runTest(i);
        }

        System.out.printf("Value: %d\n", globalRT.initValue);
    }

    RecursionTest(int i) {
        initValue = i;
    }

    public int runTest(int returnValue) {
        RecursionTest rt0 = new RecursionTest(0);
        RecursionTest rt1 = new RecursionTest(1);
        RecursionTest rt2 = new RecursionTest(2);
        RecursionTest rt3 = new RecursionTest(3);

        rotateEscape(3, rt0, rt1, rt2, rt3);

        returnValue = returnValue + rt0.initValue;
        returnValue = returnValue + rt1.initValue;
        returnValue = returnValue + rt2.initValue;
        returnValue = returnValue + rt3.initValue;

        return returnValue;
    }

    public static void rotateEscape(int callCount, RecursionTest rt0, RecursionTest rt1, RecursionTest rt2, RecursionTest rt3) {
        callCount = callCount - 1;

        globalRT = rt0;

        if (callCount > 0) {
            rotateEscape(callCount, rt1, rt2, rt3, rt0);
        }
    }
}

Output:

java -Xint RecursionTest
Value: 2

java -Xjit:"optlevel=hot,count=10,disableasynccompilation,limit={*runTest*},dontinline={*rotateEscape*}" -XX:-EnableHCR -Xshareclasses:none -Xnoaot RecursionTest
Value: -562960

Tracefile log also confirms that EA stack allocated the original rt2 when it shouldn't have.

A potential fix is to check that the parameters do not change on each layer of recursion. With more thought, perhaps a less restrictive check for ensuring an object does not escape during recursion can be determined.

[IBMJimmyk]

@hzongaro You were interested in this.