transformers

transformers copied to clipboard

Add sudachi and jumanpp tokenizers for bert_japanese

3

# What does this PR do? This PR adds a classes to use [sudachi](https://github.com/WorksApplications/SudachiPy) and [jumanpp](https://github.com/ku-nlp/pyknp) with BertJapaneseTokenizer. As a background, there are traditionally multiple tokenizers in Japanese language processing,...

r-terada

Add more model resources

7

A continuation of #19767 to add existing official resources (blog posts, notebooks, scripts, etc.) directly to the model docs for 20 of the most popular architectures based on last month's...

stevhliu

[WIP] Fix layernorm eps

2

# What does this PR do? **(No breaking change in this PR)** **(So far I only change `OneFormerConfig`, but I will update other config classes whose default `layer_norm_eps = 1e-5`)**...

ydshieh

Generate: TF `compute_transition_scores`

1

# What does this PR do? This PR adds the TF `compute_transition_scores`, akin to PT's #21191. What seemingly started off as a simple task, ended up being a complex task...

gante

Automated compatible models list for task guides

1

# What does this PR do? This PR adds a script that aggregates model architectures compatible with a task illustrated in a task guide and adds a list of links...

MKhalusova

Tf timestamps whisper + update generate support

7

# What does this PR This PR updates the way we generation TF and FLAX to fix the breaking changes that we had. It also adds support for the timestamps...

ArthurZucker

Not able to run evaluate on whisper.tflite that got generated from TFWhisper model

45

### Model description @gante Generated HF TFwhisper model into whisper.tfllite model. However, I'm not sure how to evaluate the created whisper tflite model. https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/tflite_from_huggingface_whisper.ipynb I would appreciate your assistance in...

nyadla-sys

[WIP] RWKV4Neo the RNN and GPT Hybrid Model

12

# What does this PR do? Adds the model from issue Fixes # (https://github.com/huggingface/transformers/issues/20737) ## Before submitting - [ ] This PR fixes a typo or improves the docs (you...

ArEnSc

[Time-Series] informer model

5

# What does this PR do? Adding Time Series Informer model https://arxiv.org/abs/2012.07436 Related issue: #20903 @kashif :)

elisim

add flax whisper implementation

24

Adds Flax whisper implementations, and adjusts flax generation utils to support it. @ydshieh @ArthurZucker See discussion in #19512

andyehrenberg