<Trans> can't match id in messages.mjs if the id contain emoji

[MatteoWebDeveloper]

trafficstars

Describe the bug When I use <Trans> macro with a content that contain emoji and lingui/cli to extract it. <Trans> It will not be able to match with the content in the messages but i18n._("content 🇬🇧") it will be able to

To Reproduce Steps to reproduce the behavior, possibly with minimal code sample, e.g:

import { Trans } from "@lingui/macro";
import { I18nProvider, useLingui } from "@lingui/react";
import { i18n } from "@lingui/core";
import { it, en } from "make-plural/plurals";

const LOCALE_IT = "it";
const LOCALE_EN = "en";

i18n.loadLocaleData(LOCALE_IT, { plurals: it });
i18n.loadLocaleData(LOCALE_EN, { plurals: en });

async function activateLocale(locale) {
  const { messages } = await import(`../../locales/${locale}/messages.mjs`);
  i18n.load(locale, messages);
  i18n.activate(locale);
}

activateLocale(LOCALE_IT);

function JsLingui() {
  const { i18n } = useLingui();

  return (
    <div>
      current locale: {i18n.locale}.<br />{" "}
      <button onClick={() => activateLocale(LOCALE_EN)}>change to en</button>{" "}
      <button onClick={() => activateLocale(LOCALE_IT)}>change to it</button>
      <br />
      <br />
      <Trans>Hello UK 🇬🇧 Vite + React + LinguiJS!</Trans>
      <br />
      {i18n._("Hello UK 🇬🇧 Vite + React + LinguiJS!")}
    </div>
  );
}

export function Internationalisation() {
  return (
      <I18nProvider i18n={i18n}>
        <JsLingui />
      </I18nProvider>
    </>
  );
}

first render

when I click button

Expected behavior <Trans> component should render correct translation content as this other option i18n._()

Additional context Add any other context about the problem here.

• jsLingui version 3.13.3
• Babel version 7.18.2
• Used this setup https://github.com/skovhus/vite-lingui

[INCHMAN1900]

Can you post your babel config? There should be a macro plugin since Trans is imported from @lingui/macro.

[MatteoWebDeveloper]

Yes I can share the config below, if you wish I can create a zip with the whole codebase. It was an investigation project to see if vite could work for us and with linguijs

.babelrc

{
  "presets": ["@babel/preset-react"],
  "plugins": ["macros"]
}

vite.config.js

import path from "path";
import { defineConfig, loadEnv } from "vite";
import macrosPlugin from "vite-plugin-babel-macros";
import preact from "@preact/preset-vite";
import graphql from "@rollup/plugin-graphql";
import svgr from "vite-plugin-svgr";
import { visualizer } from "rollup-plugin-visualizer";

export default defineConfig(({ mode }) => {
  const env = loadEnv(mode, process.cwd());

  return {
    plugins: [
      macrosPlugin(),
      preact(),
      graphql(),
      svgr(),
      visualizer({ open: true }),
    ],
    server: {
      https: true,
      proxy: {
        "/futurama": "https://api.sampleapis.com/",
      },
    },
    resolve: {
      alias: {
        "~": path.resolve(__dirname, "src"),
      },
    },
    define: {
      process: {
        env: {
          ...env,
        },
      },
    },
    build: {
      rollupOptions: {
        output: {
          assetFileNames: `[name].[hash].[extname]`, // this is needed only for UKC because we use absolute path but in the future UKC will be fix
        },
      },
    },
  };
});

[INCHMAN1901]

It's the emoji that causes the problem.

Trans is compiled to

_jsxDEV(Trans, {
  id: "Hello UK \\uD83C\\uDDEC\\uD83C\\uDDE7 Vite + React + LinguiJS!"
}, void 0, false, {
  fileName: _jsxFileName,
  lineNumber: 33,
  columnNumber: 7
}, this)

but i18n._ is compiled to Hello UK \u{1F1EC}\u{1F1E7} Vite + React + LinguiJS!.

The id in Trans is not included in messages file, so it's not translated.

[MatteoWebDeveloper]

Oh Yes I am aware it breaks only with emoji 👍🏻, I thought to report this bug in case it was something you want to fix.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[timofei-iatsenko]

@MatteoWebDeveloper could you also share translation files used in the example?

[MatteoWebDeveloper]

@thekip I used the js lingui cli automatic extraction

PO file en

msgid ""
msgstr ""
"POT-Creation-Date: 2019-10-17 17:15+0100\n"
"Mime-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"X-Generator: @lingui/cli\n"
"Language: en\n"
"Project-Id-Version: \n"
"Report-Msgid-Bugs-To: \n"
"PO-Revision-Date: \n"
"Last-Translator: \n"
"Language-Team: \n"
"Plural-Forms: \n"

#: src/components/translation-playground/index.jsx:30
msgid "Hello UK 🇬🇧 Vite + React + LinguiJS!"
msgstr "Hello UK 🇬🇧 Vite + React + LinguiJS!"

PO file it

msgid ""
msgstr ""
"POT-Creation-Date: 2019-10-17 17:15+0100\n"
"Mime-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"X-Generator: @lingui/cli\n"
"Language: en\n"
"Project-Id-Version: \n"
"Report-Msgid-Bugs-To: \n"
"PO-Revision-Date: \n"
"Last-Translator: \n"
"Language-Team: \n"
"Plural-Forms: \n"

#: src/components/translation-playground/index.jsx:30
msgid "Hello UK 🇬🇧 Vite + React + LinguiJS!"
msgstr "Ciao italia 🇮🇹 Vite + React + LinguiJS!"

[timofei-iatsenko]

Ok, got this. This is not an issue, it's just the way how it works.

1. {i18n._("Hello UK 🇬🇧 Vite + React + LinguiJS!")} Is non-macro call. To be picked up by extractor it should be prefixed with /*i18n*/ comment. That way you give two entries in message catalog.
2. Strings processed by macro got unicode characters replaced (escaped) for the better compatibility with the rest of pipelines (there are arabic, Vietnamese and other languages which extensively uses unicode characters). When you call i18n._ directly, you opt-out from this optimization and get another source string.
3. It's better to use macro everywhere, it would give you more consistent results. If you write
   t`Hello UK 🇬🇧 Vite + React + LinguiJS!` that would work.

[MatteoWebDeveloper]

@thekip there is a misunderstanding, i18n._ is working. It's <trans> with emoji that does not work. If you look at the first comment you will see both option displayed side by side. First string uses Trans, second i18n._. I don't mind this issue being close as we workaround this issue and it's pretty edge case.

[timofei-iatsenko]

Yes, i misunderstood the problem. I tried it in one more time just added:

<Trans>Hello UK 🇬🇧 Vite + React + LinguiJS!</Trans>

than extracted, translated and run app. And I wasn't able to reproduce it. I didn't use it with vite, just regular CRA, may be vite is the key?

[timofei-iatsenko]

Or it was fixed in latest main...

[MatteoWebDeveloper]

@thekip right now I am creating a public repo to recreate the issue. It's also possible that things has changed in 6 months.

[MatteoWebDeveloper]

@thekip I created a new project with latest react, vite, lingui and then tried with preact as well. Everything works fine, it may be the issue got resolved in between versions.

commit react setup https://github.com/MatteoWebDeveloper/internationalisation-js-lingui/commit/eaca09c77ff9fd70a8d05e2f56a15982904118af commit preact setup https://github.com/MatteoWebDeveloper/internationalisation-js-lingui/commit/d0a106fdaa59cac2963e3e63ef0b89dce7bb8e77

Thank you for listening and reaching out @thekip and thank you for this great library.