spacy-transformers

spacy-transformers copied to clipboard

Support offset mapping alignment for fast tokenizers

1

Switch to offset mapping-based alignment for fast tokenizers. With this change, slow vs. fast tokenizers will not give identical results with `spacy-transformers`. Additional modifications: * Update package setup for cython...

adrianeboyd

Transformer: add update_listeners_in_predict option

1

**Draft:** still needs docs, but I first wanted to discuss this proposal. The output of a transformer is passed through in two different ways: - Prediction: the data is passed...

danieldk

Preserve `tokenizer.is_fast` as `use_fast` in `HFShim` in order to be able to restore the same slow/fast tokenizer settings. This will be necessary when fast tokenizers switch to offset alignments from...

adrianeboyd

The output of a transformer is passed through in two different ways: - Prediction: the data is passed through the `Doc._.trf_data` attribute. - Training: the data is broadcast directly to...

danieldk

Extend support to transformers v4.39

2

## Description Extend support to transformers v4.39. ### Types of change ? ## Checklist - [x] I confirm that I have the right to submit this contribution under the project's...

hume-brian

Set transformers upper bound to < 4.42.0

2

## Description Relax the upper bound a little. ### Types of change Maintenance ## Checklist - [x] I confirm that I have the right to submit this contribution under the...

danieldk