JSON support

[JoernT]

JSON shall be supported in the same way as XML.

Despite and contradicting my former belief that support for JSON means using something like JSONPath to address data there might be a better solution using XPath JSON support in form of: parse-json()function.

This would yield to binding expressions like so: parse-json(/data)?property

or for:

<xf-instance>
<data>
{
   "a": 1, 
   "b": [3,6,9], 
   "LN": "Rielos", 
   "FN": "Lotte", 
   "IsFemale":true
}
</data>
</instance>

parse-json(/data)?a will output '1'

parse-json(/data)?b?2 will output '6'

Remains food for thought how XPath JSON support can be exactly be used for the purpose.

Avoiding an additional JSONPath dependency which also comes with its own syntax (for good and bad maybe) reduces the overall package weight by at least 250K of script.

tbc.

[JoernT]

Beyond accessing simple pathes in JSON data a filtering mechanism is essential.

Use Case: filtering lists where one depends on the other: instance('countries')/country[@continent = instance('default')/continent]

How can we express that in XQuery or XPath - in the absense of predicates in map operators something like fn:filter would be needed but with a tighter syntax?

Seeking ideas... @line-o ?

[line-o]

<fx-model>
<fx-instance id="countries" as-json>
<data>
[
  {"code": "en", "name": "England"}, 
  {"code": "de", "name": "Germany"}, 
  {"code": "au", "name": "Australia"}
]
</data>
</fx-instance>
<fx-instance id="continents" as-json>
<data>
[
  { "name": "Australia", "countries": ["au"]},
  { "name": "Europe", "countries": ["en", "de"]}
]
</data>
</fx-instance>
</fx-model>

[line-o]

If you know which continent to match against

instance("countries")?*[?code = instance("continents")?2?countries?*]

Variable continent ($selected-continent)

instance("countries")?*[?code = instance("continents")?($selected-continent)?countries?*]

[line-o]

About JSON instance data encodings

with attribute on data element

<fx-instance id="countries">
  <data as-json>
    [
      {"code": "en", "name": "England"}, 
      {"code": "de", "name": "Germany"}, 
      {"code": "au", "name": "Australia"}
    ]
  </data>
</fx-instance>

with attribute on fx-instance and without data element

<fx-instance id="countries" as-json>
    [
      {"code": "en", "name": "England"}, 
      {"code": "de", "name": "Germany"}, 
      {"code": "au", "name": "Australia"}
    ]
</fx-instance>

[JoernT]

now that the discussion started i see more options:

1. use native XPath as above
2. convert JSON instances to XML and back and keep on addressing with XPath locationPathes
3. use XPath locationPathes and 'rewrite' them to native XPath syntax and map operator (again like above)
4. use JSONPath Plus lib

[line-o]

Regarding converting instances in different formats to XML and back there is recent activity https://www.w3.org/community/ixml/

[JoernT]

after a talk with Martin i guess we came to the conclusion to rule out option 2 as this involves too many ambiguities.

so we're down to:

1. native XPath/XQuery
2. rewrite locationPathes for easier authoring
3. use JSONPath

it would need more examples to find out if 1. could be the one ;)

@DrRataplan i missed to copy your example with 'satisfies' unfortunately - could you be so kind to add this to this ticket please?

[DrRataplan]

Hey Joern, Juri,

Correct, I believe the option 2 can get very annoying to maintain and support in any sensible way. When you consider just maps and arrays, a deterministic and logical way to convert JSON to XML is easy to do. However, when you get into mixing arrays and maps on the same level, things get out of hand.

The example with quantified expressions I had earlier was something in the lines of some $code in instance("countries")?*?code satisfies $code = instance("continents")?($selected-continent)?countries to get a boolean. Looking at @line-o's example, I now see this is not fully compatible. We can consider array:filter here:

XPath playground link

(please disregard the wrong output, I seem to have a bug there, the filter syntax is not being applied for arrays. Working on that.)

Regards,

Martin

[line-o]

@DrRataplan Yes there is something going on with predicates not being evaluated.

another playground link

But that then also creates additional problems, if the sequence with predicate is a value in a function type

array to sequence with predicate as value in function type

Nice Playground BTW :)

[DrRataplan]

Hey @line-o,

Thanks! The playground is getting more visitors than our marketing website. Not sure whether the marketing peeps like it, but it's proving its usefulness!

I fixed that bug with predicates combined with lookups not being evaluated as you'd expect. Having worked with these lookups, I learned to like them. Since it would prevent a new dependency, I'd say let's go ahead with the native XPath/XQuery 3.1 way of handling arrays/maps. We can always add a new lib if we find out it is cumbersome in more complex cases. See progress at link.

[JoernT]

would like to include a basic example into 1.0.0

[DrRataplan]

Noting down another approach that may be viable:

Imagine an XPath query resulting in an JSON object. From that example @line-o made:

instance:

    [
      {"code": "en", "name": "England"}, 
      {"code": "de", "name": "Germany"}, 
      {"code": "au", "name": "Australia"}
    ]

At some point there is a query ?*[?code="en"] (playground)

Ideally, we'd like a control that just controls over "name": <fx-control ref="?name"/>. However, because that ?name is just going to resolve to England, we cannot just edit that string and expect it to magically change the instance.

Illustration in JS:

const context = instance.find(c=>c.code === 'en');
let ref = context.name;

// The following just writes to the `ref` variable: the context is unaffected.
ref = 'Italy';

However, in JavaScript, you'd do

const context = instance.find(c=>c.code === 'en');

// Note this DOES change the instance!
context["name"] = 'Italy';

Bringing this back to XForms, if we can invent a new type of control specific to JSON that accepts a ref (which results in some JSON Object / Array) plus a key to write to, we CAN write to JSON! For example <fx-json-control ref="." key="name"/>.

One 'bear on the road' though: FontoXPath right now does not keep reference equality in check when roundtripping maps and arrays. This is up for debate at https://github.com/FontoXML/fontoxpath/discussions/392. I just implemented a draft PR for it.

@JoernT @line-o what are your thoughts?

Future work: dependency resolution -> making required work in a sensible way.

[JoernT]

@DrRataplan if i got you right we can probably make use of the modelItem path property. It usually just does a path(contextnode) and saves that as a pointer in the modelItem. Wouldn't that be an option to store the value of the key?

Each control has a modelItem so has access to that. Currently these pathes are instance-absolute and with json that approach could probably be similar -> a function jsonPath() that returns a canonical path representation like e.g.

?automobiles?1?maker

Open up e.g. 02-refs.html and if it has bound nodes you should see the modelItems array logged at the end of recalculate(). Not sure myself right now but guess the path is only used by DepGraph to avoid duplicates.

I'm not a particular fan of creating another control type when using json - i mean, i don't like to introduce a new custom element for that. If necessary i would rather like to build that knowledge intro the existing one. We already have a flag on the instance and we could know about which instance a control is binding to. That way we can just ask the modelItem to which instance we're bound and act appropriately wether we're bound to XML or JSON. For the part of updating the json that could be handled within the modelItem itself (not the control).

Just noticed that i'm not having the instance directly within modelItem but shouldn't be too hard to get it in at construction time of a modelItem. Then if would be even easier to know about the type of the binding context (xml or json) and use that path to do the updating of that instance.

How does that sound?

[DrRataplan]

That sounds a lot like functional lenses: https://en.wikibooks.org/wiki/Haskell/Lenses_and_functional_references. I could try to compile XPath map / array lookups to generate a getter / setter combination to the item it would normally return. I will take some time to prototype this (outside of fontoxpath), to see how it will look.

If we have those lenses, we should even be able to compose them, as in when repeating over an array of person objects, when the name setter of a person is called, it could in turn call the setter of that person, which will in turn generate a new array that contains the other person objects, plus the new person replaced. Effectively changing the name of a person in an array of persons.

This sounds like the proper functional way to approach this problem. What do you reckon?

[JoernT]

to be honest i don't get what lenses are actually from the above link. The Haskell syntax is nothing i read easily ;) But i got the idea from another source at medium.

Essentially the 'immutable' array or map as we know it in XQuery ... Just wondering how the impact on performance is when re-generating the arrays all the time. But i guess only trying will give an answer.

All in all it's at least an approach that could work. Thanks for sharing @DrRataplan - lets talk about it coming week.

[DrRataplan]

Some progress: I just created drrataplan/xpath-lenses, which can transform XPath expressions in those lenses. This seems to work quite nicely (for simple XPaths like a?b?2).

I did not implement anything complex like functions (yet). I think I have a way though that does not require me to restructure fontoxpath fully.

Lets take some time next week to try to integrate this into Fore. I think this can already be a great first step (even though there are big parts missing, like functions, proper filters, etcetera).

[JoernT]

That's wonderful progress. Excited to see how far we can get with this approach but looks very promising now that some groundworks have been done.