Translations are dissapearing in umbrella

[amatalai]

When you run mix gettext.extract from umbrella root not every module is forced to recompile, which causes translations to disappear.

Example repo: https://github.com/amatalai/gettext_umbrella_issue

Probably somehow related to #165

[amatalai]

My current workaround: https://gist.github.com/amatalai/1877f48a195f578682cfd20e66c6544e

[josevalim]

Thanks @amatalai!

Indeed gettext does not work across dependencies. One possibly solution to this problem is for us to add an option that specifies exactly which apps to run to extract the dependencies from. Although your work around is also fine for now (albeit slightly inefficient).

[maennchen]

@josevalim I'm having the same issue as well. Was there anything done in the meantime that should solve the issue?

[josevalim]

@amatalai's workaround seems to be the way to go for now.

[maennchen]

@josevalim Could the workaround be incorporated into the extract mix task directly?

(Via Arg / Config / somehow detect the umbrella)

[maennchen]

(As always I'd be happy to provide a PR :smiley: if that is welcome)

[josevalim]

I think the work around can be more efficient but I didn't have time to look at it yet, unfortunately.

[maennchen]

Using the workaround produces a few issues:

• All translations are marked with the flag elixir-format
• No longer used translations are not discarded
• I get the following warning multiple times when extracting:

Compiling lib/hygeia_gettext.ex (it's taking more than 10s)
warning: the Gettext backend HygeiaGettext has the same :priv directory as HygeiaGettext, which means they will override each other. Please set the :priv option to different directories or use Gettext inside each backend
  (elixir 1.12.1) lib/enum.ex:2356: Enum."-reduce/3-lists^foldl/2-0-"/3
  (gettext 0.18.2) lib/gettext/extractor.ex:80: Gettext.Extractor.pot_files/2
  (gettext 0.18.2) lib/mix/tasks/gettext.extract.ex:55: Mix.Tasks.Gettext.Extract.extract/2
  (gettext 0.18.2) lib/mix/tasks/gettext.extract.ex:37: Mix.Tasks.Gettext.Extract.run/1
  (mix 1.12.1) lib/mix/task.ex:394: anonymous fn/3 in Mix.Task.run_task/3
  (mix 1.12.1) lib/mix/project.ex:353: Mix.Project.in_project/4
  (elixir 1.12.1) lib/file.ex:1560: File.cd!/2
  (mix 1.12.1) lib/mix/task.ex:520: anonymous fn/4 in Mix.Task.recur/1
  (elixir 1.12.1) lib/enum.ex:2356: Enum."-reduce/3-lists^foldl/2-0-"/3
  (mix 1.12.1) lib/mix/task.ex:519: Mix.Task.recur/1
  (mix 1.12.1) lib/mix/project_stack.ex:181: Mix.ProjectStack.recur/1
  (mix 1.12.1) lib/mix/task.ex:394: anonymous fn/3 in Mix.Task.run_task/3
  (mix 1.12.1) lib/mix/cli.ex:84: Mix.CLI.run_task/2
  (elixir 1.12.1) src/elixir_compiler.erl:75: :elixir_compiler.dispatch/4
  (elixir 1.12.1) src/elixir_compiler.erl:60: :elixir_compiler.compile/3
  (elixir 1.12.1) src/elixir_lexical.erl:15: :elixir_lexical.run/3
  (elixir 1.12.1) src/elixir_compiler.erl:18: :elixir_compiler.quoted/3
  (elixir 1.12.1) lib/code.ex:1261: Code.require_file/2

All of this can be reproduced when using mix gettext.extract.umbrella in https://github.com/jshmrtn/hygeia

[josevalim]

Hi @maennchen, I will take some time this week to look at this. Can you please make sure those problems happen in the initial repo: https://github.com/amatalai/gettext_umbrella_issue - feel free to fork it. Looking at a minimal app is going to make it a bit easier for me to address it. :)

[maennchen]

@josevalim I'll have a look if I can reproduce it there. :+1:

[josevalim]

@maennchen I noticed this fell through my cracks. The offer still stands if you are game. :)

[maennchen]

@josevalim it also fell through my cracks. I'll have a look next week since I'm currently on vacation.

[josevalim]

Enjoy the holidays!

[maennchen]

@josevalim I'm having trouble reproducing it in a minimal repo.

The things that are "special" about hygeia could be:

• Long compilation times: ex_cldr / cadastre / apps/hygeia/lib/hygeia/ecto_type/noga.ex take quite some time to compile
  • Also there's a bit of a mess in the compilation that I should probably clean up since it takes much too long overall
  • For example, a lot of changes cause basically everything to recompile
• The umbrella apps are depending on each other a lot, especially on the gettext app

Do you have an idea how I could reproduce that in a minimal repo?

The Hygeia repo itself should demonstrate the issue nicely, other than it compiling very slowly, it should demonstrate the issue.

[josevalim]

I assume you tried to reproduce from the bottom up. What if you try from the top down? I.e. by getting the current app and stripping as much code and deps as you can?

[maennchen]

@josevalim I haven't tried that yet. Mainly because it takes forever with the current compilation times.

But there's probably no way around it than to do it in reverse. I'll give it a try.

[maennchen]

@josevalim I was able to reduce it a lot. It for sure has nothing to do with the compilation time or any of the apps...

I removed so much that I can't see anymore how this would be different from any other project except the gettext.extract.umbrella mix task.

https://github.com/maennchen/elixir-gettext-issue-178

To Reproduce

• Use Elixir / Erlang Versions as written in .tool-versions (ASDF)
• Run mix deps.get
• Run mix gettext.extract.umbrella twice

Problems

Warning

The following warning should not be logged:

warning: the Gettext backend HygeiaGettext has the same :priv directory as HygeiaGettext, which means they will override each other. Please set the :priv option to different directories on use Gettext inside each backend
  (elixir 1.12.1) lib/enum.ex:2356: Enum."-reduce/3-lists^foldl/2-0-"/3
  (gettext 0.18.2) lib/gettext/extractor.ex:80: Gettext.Extractor.pot_files/2
  (gettext 0.18.2) lib/mix/tasks/gettext.extract.ex:55: Mix.Tasks.Gettext.Extract.extract/2
  (gettext 0.18.2) lib/mix/tasks/gettext.extract.ex:37: Mix.Tasks.Gettext.Extract.run/1
  (mix 1.12.1) lib/mix/task.ex:394: anonymous fn/3 in Mix.Task.run_task/3
  (mix 1.12.1) lib/mix/project.ex:353: Mix.Project.in_project/4

elixir-format

The following translation is appended to the default.pot:

#, elixir-format
#: lib/hygeia_gettext/test.ex:4 lib/hygeia_gettext/test.ex:4
msgid "Test!"
msgstr ""

Expected would be without the elixir-format flag

Not removed

No longer needed translations are not removed. This forces me to do the following in the full hygeia application:

• Manually remove everything from the pot except the header
• Delete files in _build (rm -rf _build/dev/lib/hygeia*)
• Extract Twice using mix gettext.extract.umbrella (The first time somehow no translations are added)

[maennchen]

@josevalim Is there anything else I can do to narrow down the issue?

[josevalim]

Hi @maennchen! Thanks for isolating it. I will probably have time to look at this in a week or two, meanwhile it will be sitting on my inbox, so I will get to it as soon as I can. :)

[maennchen]

@josevalim Thanks & no hurry, it's not super urgent for me. If I can be of further help, just let me know :)

[josevalim]

Folks, since you are active on the repo and I have been unable to deliver, I am leaving this in your capable hands. If you need any help or rubber ducking, I will be glad to be involved and I will still follow the issue. Sorry and thanks!

[Gigitsu]

I've made this task that seems to work. You can call it in this way:

mix gg.gettext.extract --backend app_with_gettext_backend --merge --for app_1

In this example, app_with_gettext_backend contains the module AppWithGettextBackend.Gettext, while app_1 depends on app_with_gettext_backend and uses AppWithGettextBackend.Gettext.

The task creates apps/app_1/priv/gettext/default.pot with messages found in both app_1 and app_with_gettext_backend, but it could be easily modified to generate POT files inside apps/app_with_gettext_backend/priv/gettext instead and to include more than one app.

I can create a sample project with this task if you think it would be useful