transformers icon indicating copy to clipboard operation
transformers copied to clipboard

Fix flax whisper tokenizer bug

Open hannan72 opened this issue 5 months ago • 1 comments

What does this PR do?

Fixes Bug when using whisper tokenizer for flax whisper model, according to the issue #32936

Before submitting

  • [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • [ ] Did you read the contributor guideline, Pull Request section?
  • [ ] Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
  • [ ] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
  • [ ] Did you write any new necessary tests?

Who can review?

@ArthurZucker @sanchit-gandhi

Models:

  • text models: @ArthurZucker
  • vision models: @amyeroberts
  • speech models: @sanchit-gandhi
  • graph models: @clefourrier

Library:

  • flax: @sanchit-gandhi
  • generate: @zucchini-nlp (visual-language models) or @gante (all others)
  • pipelines: @Narsil
  • tensorflow: @gante and @Rocketknight1
  • tokenizers: @ArthurZucker
  • trainer: @muellerzr and @SunMarc

Integrations:

  • deepspeed: HF Trainer/Accelerate: @muellerzr
  • ray/raytune: @richardliaw, @amogkam
  • Big Model Inference: @SunMarc
  • quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

Maintained examples (not research project or legacy):

  • Flax: @sanchit-gandhi
  • PyTorch: See Models above and tag the person corresponding to the modality of the example.
  • TensorFlow: @Rocketknight1

-->

hannan72 avatar Aug 27 '24 19:08 hannan72