infinity icon indicating copy to clipboard operation
infinity copied to clipboard

Fix BetterTransformer compatibility with transformers >= 4.49

Open AimoreRRD opened this issue 3 months ago • 1 comments

Summary

• Handle RuntimeError when importing BetterTransformer with transformers >= 4.49 • Gracefully disable BetterTransformer optimization when unavailable due to version incompatibility • Maintain backward compatibility with older transformers versions

Problem

BetterTransformer from optimum v1.27.0 is deprecated and incompatible with transformers >= 4.49, causing a RuntimeError that prevents infinity_emb from starting.

Solution

  • Wrap BetterTransformer imports in try-except blocks
  • Add BETTERTRANSFORMER_AVAILABLE flag to track availability
  • Skip BetterTransformer optimization when not available, logging info message
  • Continue normal operation without optimization

Test plan

  • [x] Test with transformers 4.53.3 and optimum 1.27.0
  • [x] Verify infinity_emb starts successfully with Qwen3 models
  • [x] Confirm graceful fallback when BetterTransformer unavailable
  • [x] Check that existing functionality remains unchanged

AimoreRRD avatar Sep 18 '25 16:09 AimoreRRD

Fixes #642

AimoreRRD avatar Sep 18 '25 16:09 AimoreRRD