Tracking file path references for each key

[emmagamma]

(I opened a related issue for the i18next-gettext-converter here: https://github.com/i18next/i18next-gettext-converter/issues/55)

tldr:

Wanting to add the functionality to track file path references to each key for the purpose of having those comments end up in .po files for our translators to get better context as-needed, requiring modifications both in this repo and in the i18next-gettext-converter, and curious how amenable you'd be to merging that sort of thing in?

---

long form:

I've been helping with some efforts to internationalize a medium-large size Ember app, and since gettext doesn't play nice with html and handlebars, we needed these two repos to generate our translation strings and convert between .json -> .po and vice versa.

But we ran into a slight issue, our translators will need file path references for each key, as comments in the .po files (something that gettext does by default in a typical i18n workflow, except gettext has line numbers, which we won't be needing) and although the gettext-converter does have some* ability to add comments, the data structure output by this tool doesn't keep track of those file paths.

Initially I modified the i18next-parser to output something that looks more like:

{
  'some key': {
    msgstr: '',
    paths: ['app/templates/error.hbs', '../fake/path']
  }
}

Where paths is an array of each file path location that a given key is found at, in case you have the same key in multiple places in your source code.

rather than the current output:

{
  'some key': ''
}

and I modified the gettext-converter to handle both formats dynamically while converting from .json -> .po, so you can have keys with file path information and others without, in the same file.

Currently, my little demo of this working involves either symlinking my local changes or pointing our app at my own forks of these two repos, which is less than ideal. I'm hoping to get these changes merged in at some point, so in the past couple days I've gone through and modified my code to use an additional flag '-t, --track-paths', so it generates the original data structure unless you include that flag and set it to true.

To further support our workflow, since the i18next runtime still needs the original format, I also made it so that using '--track-paths' will prepend '/tmp' to the output location of the translation file but not the translation_old file. So rather than running this tool to get our .json, and then running the converter to get our .po - we are running this tool with --track-paths true to get our new .json format, then running the converter on the output /tmp/locales/en/translation.json to get our .po file, and finally running the converter a second time to go from .po -> .json so that we're back in the original format, and now back in the correct directory as well.

Ideally we'd like to merge these changes back into the repos, rather than maintaining our own forks. I'm doing my best to ensure that this is all opt-in behavior and that it doesn't affect any other workflows that people may be using. Just wanted to put the feelers out there before I get my PR's up. Any feedback is appreciated!

[karellm]

My first reaction is that this is highly specific and I'm a little reluctant (but not opposed) to complicate the code for something your project likely be the only one to use. If you think there is a use case that applies to more people, please let me know.

As to the way you structured the changes, it makes total sense to me. It definitely must be opt-in. I would probably have the parser generate both a en.json and en.po.json with the track-paths option on since it feels little more predictable than the flow you currently have (going back and forth between the two libraries).

Let me know what you think but I'm definitely willing to consider adding this if we can keep the code clean and maintainable.

[emmagamma]

I can certainly understand that reaction, although I don't think it's specific to just us. It seems to be a common thing for translators to desire file path information to see more context around the string they're translating.

I like the idea of generating both a .json and .po.json, it does seem less hacky than putting it in /tmp and relying more on the other repo. I think our initial thoughts were, why modify this repo to do what the other one can already do, but it turned out not to be that big of a change from what I had.

Thanks for the feedback! after my team reviews my code, I'll put up PRs for this and we can go from there :)

[jeremyosborne]

I'm a little late to the party but I did a google search for this specific item today. I have low to average experience setting up multiple languages in an app, but in two previous companies and in an open source project I've (sort of) contributed to, having the file references helped debug context.

In my current project it will be helpful because we're using keys-as-English-defaults (except where too complex) and having the file references will be useful for our initially ad hoc translations (English first, German second).

[karellm]

@jeremyosborne see https://github.com/i18next/i18next-parser/pull/56

[nickasd]

I thought that the parser would output at most one json file per parsed source file. If this is really the case, why would you need to store the path for each key?

[karellm]

A key can appear in multiple files. The idea was for translator to know where the key come from to look for context.

[nickasd]

Right, I forgot that multiple files can be merged together.

[eMerzh]

chiming in, but isn't this plausible with

	customValueTemplate: {
		paths: '${filePaths}',
		msgstr: '${defaultValue}',
	},

? but then there is no support for the rest of the chain atm