Unable to format data from store as JSON in javascript script handler

[rnestler]

My target is to have an endpoint that implements a mocked search from given data. My aproach was to use

• A store for the data
• A JS scriptFile which reads the data and filters it

Now my issue is, that I'm unable to get the data from the store in a format, where I can render it as JSON:

Minimal example repo

api-config.yaml

---
plugin: rest

system:
  stores:
    persons:
      preloadFile: persons.json

resources:
  - path: "/persons"
    method: get
    response:
      scriptFile: response.js

persons.json

{
  "persons": [
    {
      "id": 1,
      "firstName": "Foo",
      "lastName": "Bar"
    },
    {
      "id": 2,
      "firstName": "Test",
      "lastName": "Ing"
    }
  ]
}

response.js

var personsStore = stores.open('persons');
var persons = personsStore.load('persons');
respond()
    .withContent(persons);

While I can retrieve it in raw form from the store directly:

curl http://localhost:8080/system/store/persons/persons
[ {
  "id" : 1,
  "firstName" : "Foo",
  "lastName" : "Bar"
}, {
  "id" : 2,
  "firstName" : "Test",
  "lastName" : "Ing"
} ]

It doesn't work when using the endpoint:

$ curl http://localhost:8080/persons
[{id=1, firstName=Foo, lastName=Bar}, {id=2, firstName=Test, lastName=Ing}]

While this looks a bit like JSON it isn't: The " are missing around keys and values and it uses = instead of :.

I then tried using JSON.stringify on it (repo:

var personsStore = stores.open('persons');
var persons = personsStore.load('persons');
respond()
    .withContent(JSON.stringify(persons));

But this will then return undefined:

$ curl http://localhost:8080/persons
undefined

Switching to IMPOSTER_JS_PLUGIN=js-graal (repo) leads to an array of empty objects:

$ curl http://localhost:8080/persons
[{},{}]

Is there a way to properly convert objects from stores to JSON?

[rnestler]

Maybe https://github.com/oracle/graaljs/issues/478 is related?

[rnestler]

According to https://github.com/oracle/graaljs/issues/766 and https://github.com/oracle/graaljs/issues/757 I should be able to use JSON.stringify(Object.fromEntries(Array.from(dataMap))), but this results in an exception:

...
Caused by: org.graalvm.polyglot.PolyglotException: TypeError: (intermediate value).fromEntries is not a function
...

According to https://github.com/oracle/graaljs/issues/766 this is related to "enabled nashorn compability".

Edit: Looking at https://github.com/outofcoffee/imposter/blob/e276046bc60fb9e151107fdf5364ca54ebb73e1b/scripting/graalvm/src/main/java/io/gatehill/imposter/scripting/graalvm/service/GraalvmScriptServiceImpl.kt#L72-L73

nashorn compatibility seems enabled indeed. Is there a way to disable this?

[rnestler]

@outofcoffee I think one solution would be to get rid of the nashorn JS altogether and only rely on graaljs without the nashorn-compat property set to get good support for newer JS features. What do you think of this?

[outofcoffee]

Hi @rnestler, thanks for raising this.

tl;dr - this will do the job

Working example:

var ObjectMapper = Java.type('com.fasterxml.jackson.databind.ObjectMapper');
var serialiser = new ObjectMapper();

var personsStore = stores.open('persons');
var persons = personsStore.load('persons');

var json = serialiser.writeValueAsString(persons);

respond().withContent(json);

I created a branch (https://github.com/outofcoffee/imposter/tree/spike/graal-js-modern) with a new Graal script engine implementation, without Nashorn compatibility. Unfortunately the results were the same.

So, digging deeper, the loaded data is deserialised by Java in an ArrayList of LinkedHashMap objects. These don't seem to be understood by Graal (or Nashorn)'s JSON.stringify function.

A work-around is to make use of a Java serialiser - in this case, Jackson:

var ObjectMapper = Java.type('com.fasterxml.jackson.databind.ObjectMapper');
var serialiser = new ObjectMapper();

var personsStore = stores.open('persons');
var persons = personsStore.load('persons');

var json = serialiser.writeValueAsString(persons);

respond().withContent(json);

This produces the following output:

$ curl http://localhost:8080/persons

[{"id":1,"firstName":"Foo","lastName":"Bar"},{"id":2,"firstName":"Test","lastName":"Ing"}]

It works with Nashorn as well as Graal.

[outofcoffee]

To simplify things here, there's a new loadAsJson() function for stores in v3.39.0.

Using that, the example is now:

var personsStore = stores.open('persons');
var persons = personsStore.loadAsJson('persons');

respond().withContent(persons);

You can also use JSON.parse(persons) here if you need to manipulate the data in JavaScript.

[rnestler]

I created a branch (https://github.com/outofcoffee/imposter/tree/spike/graal-js-modern) with a new Graal script engine implementation, without Nashorn compatibility. Unfortunately the results were the same

But did the JSON.stringify(Object.fromEntries(Array.from(dataMap))) snipped also still raise the fromEntries is not a function error?

[rnestler]

The new loadAsJson() works quite nice! See https://github.com/rnestler/imposter-state-minimum-example/commit/962522a508a8cd60bfa7ab8e7b6f225992f53993

Thanks for being so super responsive on this issue!

I saw that loadAsJson() is already documented in https://docs.imposter.sh/stores/. But maybe it would make sense to document that without it, the returned data may not behave as expected due to it being ArrayList / LinkedHashMap.

[outofcoffee]

Hi @rnestler, you're most welcome :) Thank you for your contributions.

We've released v3.40.0, which has an improved js-graal plugin. This one has Nashorn compatibility turned off and new plumbing for Graal to understand how to parse objects returned from stores.

You can enable this behaviour with the following environment variable:

IMPOSTER_GRAAL_STORE_PROXY=true

Would you mind seeing if this helps with the original issue?

[rnestler]

@outofcoffee I tested again with 3.40.0:

• With loadAsJson everything works as before (https://github.com/rnestler/imposter-state-minimum-example/tree/eef11675c91f0d156ed4f87db3dbc4123c5d0e14)
• With just using load I get an error

  imposter  | Caused by: org.graalvm.polyglot.PolyglotException: TypeError: invokeMember (withContent) on io.gatehill.imposter.script.ReadWriteResponseBehaviourImpl failed due to: Cannot convert '[{id=1, firstName=Foo, lastName=Bar}, {id=2, firstName=Test, lastName=Ing}]'(language: Java, type: java.util.ArrayList) to Java type 'java.lang.String': Invalid or lossy primitive coercion.
  

  (https://github.com/rnestler/imposter-state-minimum-example/tree/c2328872814b664c8bcc38480053e997d96a9b3b)
• With load and JSON.stringify I again get the [{},{}] response. (https://github.com/rnestler/imposter-state-minimum-example/tree/b82d6e68d2685139fa49ff9ee5133adde9b35a3e)
• When I set IMPOSTER_GRAAL_STORE_PROXY=true the docker container doesn't start but errors with:

imposter  | 15:40:05 DEBUG i.g.i.s.g.s.GraalvmScriptServiceImpl - Enabling Graal store proxy
imposter  | 15:40:05 ERROR i.v.c.i.l.c.VertxIsolatedDeployer - Failed in deploying verticle
imposter  | java.lang.RuntimeException: Error registering plugin: class io.gatehill.imposter.scripting.graalvm.service.GraalvmScriptServiceImpl
imposter  | 	at io.gatehill.imposter.plugin.PluginManagerImpl.createPlugins(PluginManagerImpl.kt:105) ~[imposter-config-3.40.0.jar:?]
imposter  | 	at io.gatehill.imposter.plugin.PluginManagerImpl.startPlugins(PluginManagerImpl.kt:90) ~[imposter-config-3.40.0.jar:?]
imposter  | 	at io.gatehill.imposter.Imposter$start$1.invokeSuspend(Imposter.kt:132) ~[imposter-engine-3.40.0.jar:?]
imposter  | 	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.9.10.jar:1.9.10-release-459]
imposter  | 	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108) ~[kotlinx-coroutines-core-jvm-1.7.3.jar:?]
imposter  | 	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584) ~[kotlinx-coroutines-core-jvm-1.7.3.jar:?]
imposter  | 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793) ~[kotlinx-coroutines-core-jvm-1.7.3.jar:?]
imposter  | 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697) ~[kotlinx-coroutines-core-jvm-1.7.3.jar:?]
imposter  | 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684) ~[kotlinx-coroutines-core-jvm-1.7.3.jar:?]
imposter  | Caused by: com.google.inject.ProvisionException: Unable to provision, see the following errors:
imposter  |
imposter  | 1) [Guice/ErrorInjectingMethod]: RuntimeException: Unable to load store driver: store-inmem. Must be an installed plugin implementing StoreFactory
imposter  |   at GraalvmScriptServiceImpl.onPostInject(GraalvmScriptServiceImpl.kt:92)
imposter  |   at GraalvmScriptingModule.configure(GraalvmScriptingModule.kt:54)
imposter  |   while locating GraalvmScriptServiceImpl

https://github.com/rnestler/imposter-state-minimum-example/tree/cc3109a223616f3035282ffbe85f9c027118fbb0

[rnestler]

But it seems that with the improved js-graal plugin Object.fromEntries(Array.from(persons[0])) works.

[outofcoffee]

Hi @rnestler, thanks to your excellent repro cases, we've tracked down the bug causing the error.

Version 4.0.0 switches to GraalVM as the default JavaScript engine, and it also has the fix to the issue you raised.

In v4, the store proxy is enabled by default (though you can still disable it using the environment variable), so you should be able to run your example without having to set the environment variables for the JS implementation, store proxy or install a plugin.