Support arrays when importing or exporting

[MarcARoberge]

We are using arrays in our i18next files and it doesn't seem to be compatible with Tolgee since it can't export in json as arrays.

I am suggesting to add support of arrays when importing in exporting. Key format could be prefix.0.foo, prefix.1.foo

[JanCizmar]

Hi! Thanks for letting us know! Anyway, we are trying to avoid this kind of different key behaviour while exporting. Adding this would lead to breaking change for users who already store keys like this, because now it would be exported as object.

{
  "prefix": {
    "0": "foo",
    "1": "foo"
  }
}

So I would recommend you refactor the project removing the arrays usage, if possible.

[sebastiaanviaene]

@JanCizmar Currently this is quite blocking for me to start using Tolgee. I wanted to migrate a project where I have translations in the form of arrays of objects:

"rootKey": [
      {
        "content": [
          "content1",
          "content2"
        ],
        "title": "title1"
      },
      {
        "content": [
          "content3",
          "content4"
        ],
        "title": "title2"
      }
    ]

where key (currently in Weblate) looks like this:

rootKey[0].content[0]

Personally I don't see an issue handling it that way. The choices made in Tolgee are very limiting imo.

[JanCizmar]

Hey! Thanks for the insight! @sebastiaanviaene This makes sense. The only issue I see here is that this format rootKey[0].content[0] is not anyhow defined by JSON standards. So PHP users might complain that they would like to see this format rootKey[0]->content[0]. That's we are currently very conservative in supporting new features like this.

That said. It makes sense, and it's easy to implement. So yep. IMO, we can add this support.

Opinion @stepan662 @cyyynthia @Martin005?

[MarcARoberge]

I know other modules like next-i18next use an option returnObjects. Maybe we can have by default the behaviour turned off for PHP users.

[Martin005]

@JanCizmar I think the format rootKey[0].content[0] comes from JSONPath, but there is no standard for JSONPath 🙁

[JanCizmar]

Workaround for now can be adding a script converting the files to import to flat json with such key names and the opposite when exporting. 💪

[cyyynthia]

I'm unsure about the [n] notation, specifically because, to preserve order and keep keys in the correct order within the array, it is difficult to insert in the middle of an array if that's necessary, and deleting array entries may cause holes in the numbering sequence.

Other than this potential UX problem with naming keys, considering JSON was originally derived from ECMAScript, using the JS array notation does make sense to me (I'm not so sure what you mean regarding the lack of definition by JSON standard; dot-notation isn't standardized in RFC 8259 either). A new export setting can toggle on/off the behavior of transforming [n] into arrays as well, if someone ever needs it.

[JanCizmar]

I'm not so sure what you mean regarding the lack of definition by JSON standard; dot-notation isn't standardized in RFC 8259 either

True. It was implemented before we become concerned.

[JanCizmar]

Maybe out of topic: But... I was also thinking about another way, how tu support custom stuff in import / exports.

Motivation: A lot of companies use crazy formats and they want to have it imported in Tolgee, but we can't add everything and supporting thousands of different formats doesn't much sense to me, since we would have to add brutal dropdowns with all the options and no one would find the format they're looking for.

However, If we implemented "Import/Export" customizers, we might enable users to provide their own custom user scripts to convert flat JS objects to any file format they desire.

We can use this to run the code on the server: https://www.npmjs.com/package/isolated-vm

[cyyynthia]

I think it'd be better fitted for the CLI to act as a translation layer (pun intended) for custom formats to something understood by Tolgee, while the server focuses on supporting industry standard formats only

[JanCizmar]

@cyyynthia You're forgetting that Tolgee is also used by users who don't know how to use the CLI. They don't even have node installed on their windows machines.

Currently I am talking with I non-developer guy, who needs to get files converted to CSV. In pretty non-standard format. I can create a script for him, but It would be much easier If I was able to add this custom script directly to the platform and just tell him "Select this one from custom scripts in Tolgee and you're done".

[stepan662]

I'm in favor of helping users with creating their own "transpilers", instead of supoorting everything first class. Not sure what is better if make it part of the cli or the platform itself. The platform would be probably more user friendly, but could be a potential security risk and might lead users to requesting more "scriptable" platform. It might be cool, but I'm not sure if it isn't too early for this kind of stuff.

[cyyynthia]

You're forgetting that Tolgee is also used by users who don't know how to use the CLI. Currently I am talking with I non-developer guy, who needs to get files converted to CSV.

That's a fair point. But at the same time, if the person is non-developer they'll need to have their script made by someone else. Definitely easier to use once setup though, I guess.

The platform would be probably more user friendly, but could be a potential security risk and might lead users to requesting more "scriptable" platform.

Running untrusted code is no joke and there's an infinite amount of ways it can go wrong, even when using containers and isolation and what not. It's a very sensitive task and must be handled with extreme care, preferably multiple layers of hardening and be thoroughly reviewed. Whether this runs on the server (for obvious reasons) or in the browser during download to re-format the data on-fly (risk of credentials theft, impossibility to rely on strong Content Security Policies, etc)

While it's probably the most powerful way of tackling the problem, I'm not sure coping with the overhead it brings is worth at this time, when there are many things on the roadmap that concerns a lot more people.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[JanCizmar]

This is already released. https://github.com/tolgee/tolgee-platform/releases/tag/v3.49.0