fork with some fixes

[subes]

Hi,

thanks a lot for this library. I have forked it and rewritten/fixed some parts here: https://github.com/invesdwin/invesdwin-context-julia/tree/main/invesdwin-context-julia-parent/invesdwin-context-julia-runtime-juliacaller/src/main/java/de/invesdwin/context/julia/runtime/juliacaller/pool

• use begin/end to support multi line commands
  • need to escape newlines when sending through the socket
• Pkg.installed is deprecated, I have implemented a workaround using Pkg.dependencies()
• there was no way to read the stdout from the java side
• there was no way to react to errors (stderr) from the java side
  • though the current solution can not differentiate between warnings (e.g. deprecation) and errors sadly
• replaced org.json with jackson (better handling of array vs object)
• make it recover from more error cases (e.g. errors during "get" operations and don't close during errors in "exec" operations)

If you like you can have a look and backport some fixes. The fork also includes a testsuite for writing/reading various data types (including vectors and matrices). Though I guess https://github.com/org-arl/jajub could be a more sophisticated solution since it seems to support binary transport and does not require a socket. But still, it is good to have options to choose between. Using json/socket based communication might be more robust long term, despite being slower.

[jbytecode]

sure, thank you! send the PR and let me dive into the review. hopefully we merge the twins!

[jbytecode]

unfortunately, the link you provide is broken with a 404 error code

[subes]

Sorry, the repo was still private. Should be public now!

[jbytecode]

since these improvements are not in form of a PR I couldn't follow the changes easly so it needs time to figure out what is going on.

[subes]

no problem, I found it easier to focus on just getting it to work for the start. Also it seems you are the author of RCaller, which I love. I have integrated that here: https://github.com/invesdwin/invesdwin-context-r/tree/master/invesdwin-context-r-parent/invesdwin-context-r-runtime-rcaller/src/main/java/de/invesdwin/context/r/runtime/rcaller/pool/internal What I love about RCaller is that one can see the stdout of R in java directly below the commands that are sent from java. I wonder if we can get this working in JuliaCaller as well? Somehow the current solution I took with the stdout reader does not accomplish this properly.

This is the feature that makes debugging scripts from java side very easy and why I prefer to write scripts using RCaller initially.

[jbytecode]

yes, initial versions of RCaller were my project but lots of people have contributed over a long time. so it is now public.

you can bring JuliaCaller into some level that is similar to what RCaller has, yes this requires some time, but gradually possible.

[subes]

I now can read my script outputs by switching from print() to println() in the scripts. So that is printing properly. From what I know julia determines itself if it is running in interactive/repl mode and enables the command feedback based on that: It has a command line argument "julia -i" to enable interactive mode. But adding it to juliacaller does not seem to enable the output.

[jbytecode]

maybe it is proper to find a way that wraps stdout streams into a buffer

[subes]

Here is someone doing that approach: https://julia-users.narkive.com/dIeyqZ0b/capture-the-output-of-julia-s-console

[subes]

Though I just experimented a bit further by maybe calling println(ans) after each command. The problem is that the variable ans does not get populated when running inside a "begin ... end" block. Thus "begin a = 1; println(ans); end" and "begin a = 1 end; println(ans)" always prints "nothing". Though it might be because of the ";" which prevents printing and ans reporting in REPL/interactive mode too. Though using a newline instead of ";" makes it work in repl, but it still does not work when using it with JuliaCaller.

[subes]

Ok, I now understand why the REPL output is not happening. Before the serve method, the REPL output is printed. Though the serve method:

function serve(PORT=8000, DEBUG=true)
	server = listen(PORT)
	println("Listening JuliaCaller on port $PORT")
	while true
		try
			client = accept(server)
			if (@isdefined client)
				handle_client(server, client, DEBUG)
    			else
				close(client)
			end
		catch err
			close(client)
			close(server)
			writeln("Closed connection and server, exiting")
			break
		end
	end
end

blocks the REPL in an endless loop, so it does not itself do the read-eval-print-loop. Instead the serve methods while loop would need to handle this. Though dunno how to emulate the REPL output in the loop here.

[subes]

I gave up on the REPL stuff, I guess one has to work with println's in the scripts during development. Also it seems juliacaller is significantly faster than jajub: https://github.com/org-arl/jajub/issues/2 I had to make some modifications there as well to get to speeds near juliacaller. In the end I am also using JSON with jajub because it seems jajub did not use binary transport of arrays. Instead it just parsed the output from the console. JSON should be more robust there as well, or at least for now easer to use for me.