Review requested: {MF1, Fluent} → MF2 data model → XLIFF 2 implementations

[eemeli]

In our latest plenary meeting, we agreed to (for now) proceed with exploring both of the proposed data models. To help with this, I have since then gone ahead and implemented some tooling around the data model proposed by @zbraniecki and myself.

Reviews and comments on all of these would be warmly welcomed, and I'm sure that this will be discussed at next Monday's "MF - Extended" meeting:

• Compiler from MessageFormat 1 source to the data model
• Compiler from Fluent (ftl) source to the data model
• JavaScript runtime based on the data model, in a similar shape as an Intl.MessageFormat object might eventually have
• Specification of an XLIFF 2 extension module for MessageFormat
• Converter from the data model to XLIFF 2, using said extension module

All of these transformations are complete and lossless. For MF1 & Fluent, the current implementation lifts all internal selectors to the top level, rather than using message references.

The proposed XLIFF 2 module natively supports all features of the data model, and is documented here.   The code for all of the above is available in the mf2 branch of the messageformat/messageformat repository. The most relevant (new) packages are:

• messageformat in packages/messageformat/
• @messageformat/compiler in packages/compiler/
• @messageformat/xliff in packages/xliff/

Once you've cloned the repo and checked out the mf2 branch, you need to run these shell commands to get the above packages and their dependencies tested and built:

npm install
npm test -- packages/messageformat/
npm run build:mf2

The test suites have been extracted from the relevant MF1 & Fluent projects, and provide initial end-to-end test coverage (only about 75% complete) for all but the XLIFF layers; for that there's no ready test cases available yet. They're intentionally formatted to be easy to use when testing other systems or implementations.

[eemeli]

@romulocintra As I mentioned at today's meeting, would it be possible to get a slot at the next plenary to present my XLIFF 2 module strawman proposal?

[eemeli]

I've recently added a couple of features to the mf2 branch that might be of interest:

Automatic grammatical case/gender/plural detection & reporting

If the key values of a selector look like grammatical cases, genders, or plurals, their information is included in the formatToParts output. If the case that was selected was a default/fallback case, the output includes both the actual desired case identifier as well as the one that was actually selected. See the bottom of this test suite for examples.

This is intended to provide data for the local transformations of #160, as well as allowing for improved error/fallback handling in the formatToParts caller, as discussed in #45.

Full support for source & target language resources in XLIFF conversion

MF2 splits languages at the resource level, while XLIFF keeps source & target languages as close as possible. This means that there's a bit of a structural mismatch esp. for selectors. The converters now handle this somewhat decently both ways, allowing for a single message to use completely different selectors or patterns in different languages. See this test suite for examples.

[eemeli]

Closing as out of date.