Bug: can't co-exist with pytorch-lightning

[jordane95]

I'm trying to train a T5 model with transformers library, which requires the sentencepiece library to tokenize sentence. But when I installed it with pip install sentencepiece, I can't import the pytorch-lightning library anymore. It yields

• when sentencepiece==0.1.5,

Python 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:21) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytorch_lightning
[libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already exists in database: google/protobuf/descriptor.proto
[libprotobuf FATAL google/protobuf/descriptor.cc:1401] CHECK failed: generated_database_->Add(encoded_file_descriptor, size): 
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: generated_database_->Add(encoded_file_descriptor, size): 
Aborted (core dumped)

• when sentencepiece==0.1.8

Python 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:21) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytorch_lightning
Segmentation fault (core dumped)

I reported the same issue in pytorch-lightning issues.

[h-vetinari]

If that's not a typo, then the version you're trying to install is ancient. Current one is 0.1.96

[jordane95]

Oh! I update the version to 0.1.96 and everything works well. Thank you.

[jordane95]

Hello again, I test the bug and it appears again after a month... I'm using the newest version of both package

pytorch-lightning==1.5.10
sentencepiece==0.1.96

The bug is yielded by

Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:57:06) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytorch_lightning
Segmentation fault (core dumped)

[sergii-ivakhno-kidsloop]

I am getting the same error with these versions on Python 3.8.12

[sergii-ivakhno-kidsloop]

@h-vetinari - would there be anything else you can suggest for debugging this, thanks?

[yangky11]

I'm still getting this with sentencepiece==0.1.97 and pytorch_lightning==1.7.1

[jordane95]

Hi @yangky11 , could you try to switch the import order to see if that works?

import sentencepiece
import pytorch_lightning

[yangky11]

Hi @jordane95, yes it works! But I have to import sentencepiece even when I only want to use pytorch_lightning . I think it's not a big problem for now. Maybe the issue will be gone in future versions. Thanks!

[akihironitta]

Just out of curiosity, does anyone know how the import order of these two packages makes a difference?

[taku910]

Since it issue is already fixed, let me close this issue.

[indigoviolet]

@taku910 where was this bug fixed? I'm still seeing it with latest sentencepiece/lightning.