[BUZOKNEW-473] Adjust transformers for nemo guardrails compatibility

[amatanasov]

This repository is public. Do not put here any private DataRobot or customer's data: code, datasets, model artifacts, .etc.

Summary

nemo guardrails pypi package is not compatible with latest transformers version ( fastembed tokenizers version conflict )

Rationale

[devexp-slackbot[bot]]

The Needs Review labels were added based on the following file changes.

Team @datarobot/codegen (#scoring-code) was assigned because of changes in files:

public_dropin_environments/python311_genai/requirements.txt

Team @datarobot/custom-models (#custom-models) was assigned because of changes in files:

public_dropin_environments/python311_genai/requirements.txt
public_dropin_notebook_environments/python311_notebook/env_info.json

Team @datarobot/nbx (#dr-notebooks) was assigned because of changes in files:

public_dropin_notebook_environments/python311_notebook/env_info.json

If you think that there are some issues with ownership, please discuss with C&A domain at #core-backend-domain slack channel and create PR to update DRCODEOWNERS\CODEOWNERS file.