Make `lingui compile` support concurrent mode?

[yunsii]

Problem Description

It will generate too many po files with dependency tree crawling in a big project. So it took quite some time to compile.

Proposed Solution

Make lingui compile support concurrent mode.

Alternatives Considered

Maybe not.

Additional Context

https://deepwiki.com/search/how-many-concurrent-limit-with_5a79ea40-5634-4a22-9c71-64fed3398c60

[timofei-iatsenko]

Hello! How much is "quite some time" ? I need to understand the baseline

[yunsii]

Now we have more than 4000 po files, it spend more than 1000 seconds in local machine.

[timofei-iatsenko]

@yunsii there is a pretty simple code processing this one, it should be quite simple change to make it concurent instead of sequential, could you try doing this locally and compare the results?

I'm afraid that just making this concurrent (single core, utilizing event-loop) wouldn't be enough, what you actually want to make it parallel (running on multiple machine cores), this might be a slighlty bigger change.

The problem is that i don't have such a big project to play with, making it parallel using multiple cores may not make it faster because of overhead of spawning multiple workers, reading / compiling lingui configuration and so on. So ideally would be nice to have something big and than play with it.

(just thinking outloud, may be just generate some set of dummy catalogs for perf test...)

[yunsii]

Okey, we will do a test in our spare time.

[timofei-iatsenko]

@yunsii i added a full concurrency for extract / compile, could you compare how that helped (if helped at all)

[yunsii]

@timofei-iatsenko Could you release a pre-release version for testing?

[andrii-bodnar]

I plan to release a new version this week. In addition, there are some minor fixes waiting. It may be a regular release, just to avoid overcomplicating things with a pre-release.

[karmeleon]

Just chiming in here, for my own use case (~43 pofiles, ~6700 keys) lingui compile takes around 1000s on both 5.4.0 and 5.3.3. It seems to use a full CPU core the whole time in both cases, so I think it may need a full threading implementation to see a significant speedup.

[timofei-iatsenko]

That was my assumption as well. The extraction is CPU bound load, so there no much speedup would be in concurrent vs sequntial mode.

[timofei-iatsenko]

Full multithreading support was merged to the main branch https://github.com/lingui/js-lingui/pull/2320

My test project reports a significant performance improvements.

[karmeleon]

That branch cut our compile times by 90%, incredible! Thank you :)