RuntimeError: Already borrowed

[jroakes]

Describe the bug Transformers tokenizers has a bug (not a bug?) detailed here where truncation causes a RuntimeError: Already borrowed error when processing the across threads.

To Reproduce Here is a minimal example of the error in Colab: https://colab.research.google.com/drive/1GCr4nuasc6I2GeYbeNt6eaE6JPUcfGav#scrollTo=F2g965a5LSk_

Expected behavior Expected predictions to be thread safe.

Screenshots N/A

Environment (please complete the following information):

• Windows / Colab

Additional context Originally reported here One solution is to pass use_fast = False but this fails due to dependency in the flair code on return_offsets_mapping.

[lawrence-pressinnov]

Hi As anyone found a proper workaround to this problem? @jroakes can you give more details on use_fast = False I don't understand where to apply this (and thus can not experiment the dependency issue).

I'm stuck with this issue on production (flair(TextClassifier) + fastapi + kubernetes autoscaling GPU nodes) and nothing helps. I tried to constraint the number of workers to one on uvicorn and even this does not prevent threads to multiply (and tokenizer to be shared amond these).

I was thinking of moving to TorchServe or KServe but I'm not sure Flair's models would be compatible nor that it would solve / mitigate the problem.

Any idea / input is welcome

[ganga7445]

I know tagger.predict(batch_sentences) is causing the problem because of multiple processors. how to pass use_fast = False in flair code?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[lsmith77]

@lawrence-pressinnov did you ever get Flair working on TorchServe?

[lawrence-pressinnov]

@lsmith77 we gave up on this and removed Flair from our solution :-/ sorry for not helping