Proposal: full XPath 1.0 support

[MartijnR]

• all XPath 1.0 axes that would allow us to e.g. refer to sibling nodes in various ways (see e.g. #85)
• all XPath 1.0 functions, including (maybe more): starts-with(), ends-with(), contains(), substring-after(), substring-before(), translate(), normalize-space(), floor(), ceiling(), last(). The first ones are particularly interesting for querying (external) data using predicates
• all XPath 1.0 paths (including e.g. /path/to/@attribute and /path/to/@ns:attribute). Not sure how to argue for this yet, but I'm starting to find this very useful for custom client forks.

We've already started incorporating XPath 2 and XPath 3 features, without fully supporting XPath 1.0. It think full support for XPath 1.0 would provide a great basis for further extensions, in particular custom XPath functions as they can then be implemented by simply translating them to pure (complex) XPath 1.0 expressions. One example is extending the pulldata() function for XML data.

[lognaturel]

Why would we want to support XPath 1.0 rather than going straight to 3.0? Seems like 3.0 is a superset with some things renamed/improved. Do we need to follow the evolution for some reason?

[MartijnR]

I think it make sense to first do 1.0 as that's part of XForms 1.0 and 1.1 and some functions we already use may be deprecated in 3.0.

However, maybe a more pragmatic reason is that I think it might be easier for a client developer to start with XPath 1.0 and expand upon it, due to the wider availability of 1.0 libraries. All modern browsers and Android/iOS webviews support 1.0 natively, for example (and it's extremely fast compared to a JS library)

However, the relevance of this really depends on the actual differences of course. I haven't yet found a good resource explaining the differences between 1.0 and 3.0. Have you perhaps?

[lognaturel]

I think this is kind of converging back with #95 then. I agree that "ability to use standard off-the-shelf" components is a really important consideration. It looks like there is no XQuery support in browsers, for example. This is an interesting list of incompatibilities between 1.0 and 3.0. I will study this more and think more about this and #95 in the next several days.

[lognaturel]

By several days I meant maybe months. 😆 Little by little...

@dcbriccetti did a little looking into XPath libraries for JavaRosa at opendatakit/javarosa#17 and found that the most promising library on the Java side provides support for XPath 2.0, 3.0, and 3.1 but not XPath 1.0.

I came across this interesting discussion of XPath 2.0 which explains that XPath 2.0+ includes a subset of XQuery. When I had read that XQuery was a superset of XPath I thought it was because XQuery contained XPath but actually it's that XPath 2.0+ include subsets of XQuery. Confusing.

Another big difference between XPath 1.0 and 2.0 described effectively by that same document is an expansion in expression types supported. I think "everything is a sequence" also has pretty significant implications. Basically, it's a big change. The Wikipedia page on XPath 2.0 is also pretty good.

So it seems we'd get a lot of value going to full XPath 2.0 as part of the ODK XForms standard. Certainly, on the Java side, going first to full XPath 1.0 support would not be helpful. But @MartijnR, is there a path to XPath 2.0 support in Enketo?

[MartijnR]

Thanks for looking into this!

I'll check out the references with interest (within a few days ;)). I'm particularly interested in backwards compatibility of XPath 1.0 expressions (XForms) in XPath 2.0, and available JS libraries.

Probably, the main issue of XPath 2.0 (and 3.0) support for browser/webview based implementations, such as Enketo, will be performance. The natively supported XPath 1.0 is about a factor 100 faster than a JS implementation (depending on what you're testing of course) - so it's very significant. I don't believe the browser vendors are lining up to support new XPath versions, unfortunately.

(Very glad to hear you're considering switching to an off-the-shelf XPath library! 👍 )

pinging @alxndrsn fyi

[lognaturel]

Talked to Dimagi some more about this since they have JavaRosa experts and unfortunately their experience is that an off-the-shelf library is not fast enough on phones. We can do some experimentation but it's seeming unlikely that we can move in that direction.

[MartijnR]

Ah ok. We can perhaps split this issue up into the more exciting little (but :boom:POWER:boom: boosting) improvements. I think these:

• starts-with(), ends-with(), contains() => now done in #105
• preceding-sibling axis

[lognaturel]

Sounds good to me. There's still some experimentation and learning I think we need to do on the JavaRosa side before we can for sure confirm that we'll have to do it custom but it's seeming likely. 💥POWER💥 is good.