TypeError: cannot pickle `Language` object with v2.0.1

[tboye]

After switching to v2.0.1 from v1.3.4 I'm facing TypeError: cannot pickle 'builtins.Language' object when running something along the lines of

import copy
from lingua import Language

l = copy.copy(Language.ENGLISH)

I've checked it with Python 3.10 and Python 3.11.

Am I missing something?

[pemistahl]

Pickling does not work out-of-the-box anymore because you are dealing with a Rust object now, not a Python enum. Pickle support with PyO3 is possible but complicated, that's why I have left it out so far until people start to complain which happened quicker now than I expected.

Why do you need pickle support in the first place? If you are using the multiprocessing module, it is not needed anymore since Lingua now brings its own parallelization methods. If you want to persist Language objects to disk, have you thought about persisting their string representations instead, e.g. Language.ENGLISH.name?

[tboye]

Thanks for the explanation.

I'm using a framework that pickles its settings and one of the settings happened to be a lingua.Language object in my case. But yeah, I can work it around by having a string representation in the settings and then creating an object from it.

Thanks for a nice library!

[pemistahl]

You are welcome. It's good that you have opened this issue. I will try to implement pickle support for the next major release as it is definitely useful to have.

[vrdn-23]

I would like to see this issue cause we're looking to implement something that doesn't block an async event loop. So we get this same error when trying to put it into a ProcessPoolExecutor. Looking forward to having this ASAP! Thanks a lot and great work!

[DanielHabenicht]

We came across the same problem in combination with supplying a parameter based on lingua.Languages to python-dependency-injection

@pemistahl A similar error occurs if one wants to use normal enum conversion:

>>> from lingua import IsoCode639_1, Language
>>> IsoCode639_1["en"]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: type 'builtins.IsoCode639_1' is not subscriptable
>>> 

Should I open another one for this?

[pemistahl]

@DanielHabenicht I will fix these errors shortly. I'm currently preparing a new major release 1.4.0 for the pure Python edition of the library. After this is done, I will concentrate on the Rust edition again.

Thanks for your patience everyone. With more kids and less time than last year, further development of my projects has slowed down but not stopped.