ltp
ltp copied to clipboard
Can't Run in MultiThread ,error occour :Already borrowed
File "/ssd/worker/wyj/anaconda3/envs/tf15/lib/python3.7/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool return await loop.run_in_executor(None, func, *args) File "/ssd/worker/wyj/anaconda3/envs/tf15/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/ssd/worker/dzf/py/cjbdiai/services/ltpserver.py", line 32, in nlp seg, hidden = ltp.seg(sentences) File "/ssd/worker/wyj/anaconda3/envs/tf15/lib/python3.7/site-packages/ltp/frontend.py", line 53, in wrapper return func(*args, **kwargs) File "/ssd/worker/wyj/anaconda3/envs/tf15/lib/python3.7/site-packages/ltp/frontend.py", line 254, in seg **kwargs File "/ssd/worker/wyj/anaconda3/envs/tf15/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2456, in batch_encode_plus **kwargs, File "/ssd/worker/wyj/anaconda3/envs/tf15/lib/python3.7/site-packages/transformers/tokenization_utils_fast.py", line 382, in _batch_encode_plus pad_to_multiple_of=pad_to_multiple_of, File "/ssd/worker/wyj/anaconda3/envs/tf15/lib/python3.7/site-packages/transformers/tokenization_utils_fast.py", line 335, in set_truncation_and_padding self._tokenizer.enable_truncation(max_length, stride=stride, strategy=truncation_strategy.value) RuntimeError: Already borrowed
在导入ltp前加入这一句试试?
os.environ['TOKENIZERS_PARALLELISM'] = 'true'
您好,我在用ltp多线程处理的时候也出现了同样的报错 我在import ltp之前加上了这行代码,但是似乎没有解决问题...还是会报这个错
请问现在有解决办法了吗