Async user function support

[hendrikdevloed]

In my use case I'd like to provide an callback to an async user-defined function.

I tried running a tool https://github.com/Leoltron/CascadeAsyncifier

cd src
CascadeAsyncifier.exe --target-framework net6.0-windows7.0 --omit-async-await --starting-file-path-regex "Jsonata.Net.Native"

after seeding two "async germination points" in FunctionCall.cs and FunctionTokenCSharp.cs.

internal virtual Task<JToken> InvokeAsync(List<JToken> args, JToken? context, EvaluationEnvironment env)
    => Task.FromResult(Invoke(args, context, env));

(not correct because it should be abstract but enough to be syntactically correct and not require modifications in all derived classes other than FunctionTokenCSharp)

In FunctionTokenCSharp.cs, I made a copy/paste of Invoke that would resolve the async result from the user callback.

internal override async Task<JToken> InvokeAsync(List<JToken> args, JToken? context, EvaluationEnvironment env)
// ...
resultObj = this.m_methodInfo.Invoke(this.m_target, parameters);
if(resultObj is Task<object?> t)
{
	resultObj = await t;
}

After that I needed some manual cleaning, the seed point that got converted to AsyncAsync needs to be deleted, etc. I still have to test with my own async user function and try to actually get it to work, but in principle I think it could be feasible. I just wanted to check if the async conversion could succeed and if the performance would tank.

Running the benchmark didn't seem to take a major hit.

Question

So I thought I'd raise this experiment here as a feature vote...

Is this something that is viable to backport to the official jsonata.net.native or not? Do people want to be able to plug in async "$myfunction(args)" implementations?

I leave in the middle how the sync/async code could coexist: autogenerated (like npgsql async started a long time ago), refactored starting from the automatic conversion to remove duplication, or using some custom SourceGenerator specific for this library among the MSBuild steps.

The log of the conversion for reproducibility: jsonata.net.native.async.txt

[hendrikdevloed]

FYI, the spike so far: https://github.com/hendrikdevloed/jsonata.net.native/tree/make-async

[mikhail-barg]

Hi @hendrikdevloed! Thank you for the interest in this project and for the effort that you put into exploring this approach. Also, sorry for not answering earlier, I seem to be missing github notifications somehow (

Getting to the point of your proposal, could you please provide some more information on why would you like to use async functions in a jsonata query?

---

When I was initially implementing the port, I also considered implementing the whole processing in an async way, but there were the following reasons why I decided not to:

1st: whole processing of query parsing or evaluating it CPU-bound. It's just a straight flow of computations, so it would not benefit from splitting into acync tasks by itself.

The only reason why it could benefit from async processing would be if it is done over the async stream reading. And then here comes this:

2nd: whole processing of the query is done in a DOM-like way. So it requires having a whole parsed source json document in memory. Which effectively disables this option too.

So the only async part I could think of would be making an async version of Jsonata.Net.Native.Json.JsonParser.Parse(). Which is possible but I considered it to be of little use.

---

Well, what I totally missed in my reasoning is usage of async functions. So thanks for raising this issue. Still I have a hard time imagining a case when it would be reasonable — for the same reasons as above — i.e., what is a case when you'd need an IO-bound code for a function that is a part of json transformation?

So please share your case, if you find it reasonable.

[hendrikdevloed]

Hi, my use case is adding a way to read the state of a machine parameter (e.g. a motor speed) via a custom Jsonata function. And the call to get the speed value is unavoidably async.

We use(d) JSONata as a domain specific language that a user can type to create conditions to control steps in a machine controller.

So somehing like $speed("motor1") >= $speed("motor2") or similar, things that I can't really cache but need to query as the evaluation occurs. I simplify my use case, the expressions are more than motor speeds, but the idea stands.

In a previous (working) synchronous attempt I tried exposing the parsing tree functions from jsonata.net.native (not "internal" anymore), parsing the expression, walking the tree and picking out the $speed("motor1")-style calls, extract the string constant parameter then query these, then provide the speed results in a dictionary to the synchronous $speed() implementation when evaluating the expression synchronously. It worked but it had disadvantages:

• all parameters in an expression were always queried, even those that might not be needed (short-circuit operators)
• the parse tree needed to be of the exact form $speed("somestring"), disallowing more complex structures like $speed("motor"+$i)

In the recent months, the async patched jsonata.net.native version has worked perfectly. Unfortunately, it is no longer in use. Not because of anything related to jsonata.net.native, just because we decided to make a more strict DSL for expressions that closely matches what we allow the user to create. JSonata is pretty broad and gives too much power/responsibility in the hands of users who only need to make some basic expressions.

[mikhail-barg]

@hendrikdevloed thank you for sharing your use-case! I also believe that jsonata is not meant to be a DSL. It's just a JSON transformation language after all.

I'm closing the issue for now, feel free to reopen in case you believe there's more to it.

And thanks for your interest in the project!