backout Reland [transformer] BT enablement on fairseq - fairseq change

[frank-wei]

Before submitting

• [x] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
• [x] Did you read the contributor guideline?
• [ ] Did you make sure to update the docs?
• [ ] Did you write any new necessary tests?

What does this PR do?

Revert the PR #4513 since that PR will break the torchscript of quantized model.

PR review

Anyone in the community is free to review the PR once the tests have passed. If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[frank-wei]

@dianaml0 the test looks like flaky?

[dianaml0]

@frank-wei yes the breaking test is unrelated, fixing that separately. Could you just make sure that you use the pre-commit hook for this (run flake8 and black) before submitting?

[dianaml0]

Could you also update the PR with a link to the PR you're reverting and a quick description?

[frank-wei]

@frank-wei yes the breaking test is unrelated, fixing that separately. Could you just make sure that you use the pre-commit hook for this (run flake8 and black) before submitting?

yes I did.

[frank-wei]

Could you also update the PR with a link to the PR you're reverting and a quick description?

Updated.