Unicode support for neuron CLI

[cipharius]

Neuron currently does not accept unicode characters in it's CLI.

For example, neuron new Piemērs results in a note with title Piem��rs. Same applies to other commands, such as neuron query -t piemērs.

This problem only exists when using neuron CLI. If notes have unicode characters, they get correctly rendered in the static website.

[srid]

Re-occurence of #273 ?

[srid]

Probably something to do with your terminal encoding? If so, perhaps we can document the environment fix in neuron guide.

[cipharius]

Locale seems to be correct and other CLI tools seem to pick up on the unicode characters. I'm using kitty as terminal emulator.

Edit: Right, it seems that https://github.com/srid/neuron/issues/273 describes the same issue.

[cipharius]

Just discovered that I don't have this problem when running ./bin/run query -t piemērs in neuron source directory. I updated my binary with nix-env -if https://github.com/srid/neuron/archive/master.tar.gz and even on newest version I still have the problem with unicode characters.

[cipharius]

Alright, I figured it out, problem is with LOCALE_ARCHIVE environment variable.

Interactive sessions work because nix-shell environment has LOCALE_ARCHIVE pointing to locale archive in nix store. If I define this environment variable outside of nix-shell, neuron CLI works correctly.

This probably means that my locale-archive is not correctly generated, or that binary build with nix glibc locale archive is not compatible with my locale archive. I'm not sure, I don't understand the role of the locale-archive.

[cipharius]

Just discovered that setting LANG to C.UTF-8 fixes this problem without changing archive variable.

Again, don't know enough about locale mechanism to tell if this is a viable solution, or just a temporary one.

Ideally I wouldn't want to change my system-wide locale just for one application.

[srid]

@maralorn Is this something that can be addressed in Nix?

[cipharius]

Researching for some more shows that Arch Linux does not support C.UTF-8 locale. Also, found out that other projects have faced this issue: https://github.com/electron-userland/electron-builder/issues/2340

[srid]

Perhaps we need to add glibcLocales as a dependency? https://github.com/nh2/nixpkgs/commit/9cfb48061de1808d26f0ff48a5a94e658b2e78fd

[fiatjaf]

I'm facing this issue. I'm on Archlinux. Setting LANG to C.UTF-8fixed it. What should I do?

[srid]

Is this even a neuron issue?

[fiatjaf]

I have no idea. It only occurs in Neuron. Maybe it is a nix issue?

[TheKK]

Here's my observations (on Ubuntu 16.04.7)

• When running neuron query, it tried to read locale from glibc, /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/locale/locale-archive. This could be observed via strace
• Running localedef --list-archive /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/locale/locale-archive tells only C.utf8

AFAIK, there's no way to make /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/locale/locale-archive to contain more locale data, that would violate the principle of Nix. So I believe the only way to fix this issue is directing neuron to desired locale-archive.

According to nix dev mailing list and setup hook of glibcLocales, we could set LOCALE_ARCHIVE environment variable to achieve what we want.

And it does work!

Running LOCALE_ARCHIVE=/usr/lib/locale/locale-archive neuron query reports the correct response (that "一些中文"):

{
  "query": [
    "ZettelQuery_ZettelsByTag",
    [
      [],
      "OrdinaryConnection",
      {
        "zettelsViewGroupByTag": false,
        "zettelsViewLinkView": "LinkView_Default"
      }
    ]
  ],
  "result": [
    {
      "zettelContent": [],
      "zettelDate": [
        "2020-10-07",
        "12:45:00"
      ],
      "zettelError": {
        "Right": []
      },
      "zettelFormat": "markdown",
      "zettelID": "71462dd7",
      "zettelPath": "71462dd7.md",
      "zettelQueries": [],
      "zettelTags": [],
      "zettelTitle": "一些中文",
      "zettelTitleInBody": true,
      "zettelUnlisted": false
    }
  ],
  "skipped": {}
}

So I think there's nothing could be improved on the neuron side, this kinds of setting could only happens in the user's running environment. Maybe there's more convenience way on NixOS though, consider there's a setupHook for glibcLocales.

[srid]

So I think there's nothing could be improved on the neuron side, this kinds of setting could only happens in the user's running environment.

I think we can automate this by creating a wrapper neuron script that uses this environment variable. It can be done in flake.nix (under defaultApp) so that users don't have to do anything special after installing neuron via nix.

For the corresponding docker change, see https://github.com/srid/neuron/commit/cbdaeeed962042094303ed6e94c26f29a7bc74b1