Non-continuable method calling continuable method leads to OOM instead of not instrumented exception

[forchid]

In this test case, the iterate() method call the continuable method suspend(), then OOM error happens！ Java verison 1.8, and net.tascalate.javaflow.api-2.7.2.

The test case

import org.apache.commons.javaflow.api.*;

public class Test {
	
	static final int N = Integer.getInteger("N", 10000000);
	
	public static void main(String[] args) {
		long ts = System.currentTimeMillis();
		//Continuation co = Continuation.startWith(new Execution(), true);
		// Lambda example
		Continuation co = Continuation.startWith((CoRunnable)(() -> {
			int i = 1; 
			for (; i <= N; i++) {
                                iterate(i);
                        }
			System.out.println("i = " + i);
		}), true);
		int i = 1;
		
                for (; null != co; ) {
                        Object va = co.value();
	                //if (!va.equals(i)) throw new AssertionError(va + " != " +i);
	                //System.out.println(va + " <-> " +i);
                       co = co.resume(va);
	               ++i;
                }
		
		long te = System.currentTimeMillis();
		System.out.printf("items %d, time %dms%n", N, te - ts);
	}
	
	//@continuable
	static Object iterate(int i) {
		before();
		Object res = suspend(i);
		after();
		return res;
	}
	
	static void before() {}
	
	static void after() {}
	
	@continuable
	static Object suspend(int i) {
		return Continuation.suspend(i);
	}
	
	interface CoRunnable extends Runnable {
		@continuable void run();
	}
	
	static class Execution implements Runnable {

                @Override
                public @continuable void run() {
                       for (int i = 1; i <= N; i++) {
                               Continuation.suspend(i);
                       }
                }
        }
}

java -javaagent:D:\lib\java\javaflow.instrument-continuations.jar Test

The test result

[main] INFO org.apache.commons.javaflow.agent.core.ContinuableClassesInstrumentationAgent - Installing agent...
[main] INFO org.apache.commons.javaflow.agent.core.ContinuableClassesInstrumentationAgent - Agent was installed
i = 10000001
i = 10000001
i = 10000001
[main] ERROR org.apache.commons.javaflow.core.StackRecorder - Java heap space
java.lang.OutOfMemoryError: Java heap space
        at org.apache.commons.javaflow.core.Stack.ensurePrimitivesStackSize(Stack.java:323)
        at org.apache.commons.javaflow.core.Stack.pushInt(Stack.java:215)
        at Test.suspend(Test.java:48)
        at Test.iterate(Test.java:37)
        at Test.lambda$main$0(Test.java:16)
        at Test$$Lambda$1/1627800613.run(Unknown Source)
        at org.apache.commons.javaflow.core.StackRecorder.execute(StackRecorder.java:122)
        at org.apache.commons.javaflow.api.Continuation$SingleShotContinuation.resumeWith(Continuation.java:573)
        at org.apache.commons.javaflow.api.Continuation.resume(Continuation.java:314)
        at Test.main(Test.java:26)
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at org.apache.commons.javaflow.core.Stack.ensurePrimitivesStackSize(Stack.java:323)
        at org.apache.commons.javaflow.core.Stack.pushInt(Stack.java:215)
        at Test.suspend(Test.java:48)
        at Test.iterate(Test.java:37)
        at Test.lambda$main$0(Test.java:16)
        at Test$$Lambda$1/1627800613.run(Unknown Source)
        at org.apache.commons.javaflow.core.StackRecorder.execute(StackRecorder.java:122)
        at org.apache.commons.javaflow.api.Continuation$SingleShotContinuation.resumeWith(Continuation.java:573)
        at org.apache.commons.javaflow.api.Continuation.resume(Continuation.java:314)
        at Test.main(Test.java:26)

[vsilaev]

Is stack trace that short or is it truncated by you? Typically, in such scenario StackOverflowException occurs earlier than OutOfMemoryException...

[forchid]

Is stack trace that short or is it truncated by you? Typically, in such scenario StackOverflowException occurs earlier than OutOfMemoryException...

It's the whole stack trace.

[vsilaev]

Verified. Indeed, the code with non-continuable method calling continuable method creates an infinite loop here. You can check it if N=10 (any small number). Raising the number of iterations just reduces the time before OOM exception. I do not plan to fix this right now. Actually, I see no straightforward way to fix this for dynamic code execution, the only possible option is a static code analysis that checks for non-continuable -> continuable code calls. It implies analyzing bytecode of all methods - this what I tried to avoid when porting from JavaFlow due to inherent overhead of such approach. Anyway, I acknowledge that such analyzer is necessary and should be optionally available. However, I do not plan to add it in nearest time.

[forchid]

Actually, I see no straightforward way to fix this for dynamic code execution

Can we decide the non-continuable method in a continuable method in instrument module? If we can, add a thread-local boolean variable continuable representing whether the current method is continuable method or not. Before calling a non-continuable method except Continuation.suspend() in the continuable method, the variable continuable is set to false, and it's set true after it is called. If the variable continuable from thread-local is false in Continuation.suspend(), then throws the IllegalStateException("Continuation.suspend() is called in non-continuable method") for checking non-continuable -> continuable code calls.

Or Before calling a non-continuable method except Continuation.suspend() in the continuable method, calls the method degisterThread(), and calls registerThread() after it is called.