Clean i18n

[danielbeeke]

Start of adding support for languages.

I closed the previous try: https://github.com/LDflex/LDflex/pull/69.

This PR adds support for:

await tomato.label['@nl'] // Tomaat

const context = {
  '@language': 'de'
}
await tomato.label // Tomate

[danielbeeke]

A big challenge here is how to know to resolve to a language code:

This tomato.label.en is the syntax I would have liked the most. But there is no static way of knowing for sure you are dealing with a language code, especially inside supports. Of course the developer could give a list of possible languages, but I think that is not compatible with other ideas in LDflex where a dynamic reaction to whatever seems the content seems to be the main idea.

The current solution is to prepend a '@'. This is workable but also not perfect because the JavaScript syntax does not that inside certain notations so you fallback to: tomato.label['@en'].

Should we also add something like a language fallback chain? I mean, what if the request triple is not translated into a specific language, how will we fallback? The current solution limits flexibility here because we limit the results that are fetched via the Sparql filter. Although I do not really know how it works internally, do we still have the triples in memory that were ignored? (in the example: labels that have a different language).

A language fallback chain could look like: ['en', 'de', 'ru'].

It might be useful for situations where you have different scripts mixed, like Latin and Cyrillic. It could also automatically detect the script and prefer same scripts first.

[jeswr]

The current solution limits flexibility here because we limit the results that are fetched via the Sparql filter. Although I do not really know how it works internally, do we still have the triples in memory that were ignored?

This would be specific to the engine executing the query which we can't make any assumptions about. For instance - if it just forwards the query to a SPARQL endpoint then the answer would be no.

I like the idea though - one solution would be to introduce the concept of 'fallback queries' to LDflex, which will keep trying different queries produced by a generator until one produces the expected results.

[jeswr]

The current solution is to prepend a '@'. This is workable but also not perfect because the JavaScript syntax does not that inside certain notations so you fallback to: tomato.label['@en'].

One option is to test a list of languages that you have specified in the resolver beforehand; i.e. The test methods just dues an includes check on a list

[danielbeeke]

I have been thinking a bit generally I see two options:

Option 1, current approach

Using the SparqlHandler we filter on the specific language. If the language is not available we fire another query. Until we get what we want. This is quite hard, in some situations you would like to have any language label and not a language from a specific list. For example a book store with books in different languages. With some many languages that not every language also has a UI language.

Option 2

We fetch the data and all the languages for a predicate. This would give us a list of languages that are available. Inside resolve of AbstractPathResolver we try expandProperty, catch the error and if there is an error we try execute all things, when that is done we return a new pathData with the specific language item loaded.

My preference would be option two. It would give us:

• One query for language items, including falling back to other languages.
• Optionally (configurable) a language fallback, meaning you request @de but you get @en. This should only happen if configured.
• tomato.label.en would be one query.

[danielbeeke]

While coding option 2 a bit I found that the resolve method is not good enough. Probably we should have a language aware then handler.

That is what I coded up before, Ruben Verborgh reviewed that and said that the then handler should not be language aware but I have come to doubt that. The then handler is a perfect spot of returning a specific language item.

I could make a LanguageAwareThen that people can use in the handlers instead of the standard 'then'. But to me language should something that would always work so I think it would be logical if LanguageAwareThen is the default then.

What do you think?

[danielbeeke]

I have a LanguageHandler now also working, this allows for: await tomato.label.nl. You should add these beforehand to the PathProxy. I have also added test to ensure multiple values for one language are also working.

[danielbeeke]

At the moment having a @language inside the context limits getting results if that specific language is not available. There is no fallback mechanism available.

[danielbeeke]

Is it normal that the following does not work?

await tomato.label.nl.sparql // undefined

I would expect some value. If needed I can make a test for it. It seems to do with async* _handle but I don't know how to run pathData.extendPath that is run after the query.

[jeswr]

Is it normal that the following does not work?

await tomato.label.nl.sparql // undefined

I would expect some value. If needed I can make a test for it. It seems to do with async* _handle but I don't know how to run pathData.extendPath that is run after the query.

In the case of the resolver appears to already work, and is just missing tests - it should test both @lang in the path, and the use of a default lang; so please add these.

In the case of the handler I believe it should also work (though I'm not stuck on this point since you do have to do filtering post hoc).

[jeswr]

In light of these comments around testing could you also add some integration tests like this one which evaluates the query against an actual query engine.

[danielbeeke]

I will close this one in favor of #325