Ruhollah Majdoddin

@sgugger @lewtun Why the inputs are truncated at 128 tokens, although the model can take 512 tokens? ```python max_input_length = 128 model_inputs = tokenizer(inputs, max_length=max_input_length, truncation=True) ``` And this is...

Compilation error

6

I got this error while compiling on Debian 12 (bookworm) with kernel 6.1.0-12-amd64: /var/lib/dkms/rtl88x2bu/5.13.1/build/os_dep/linux/wifi_regd.c:409:36: error: ‘REGULATORY_IGNORE_STALE_KICKOFF’ undeclared (first usein this function) 409 | wiphy->regulatory_flags |= REGULATORY_IGNORE_STALE_KICKOFF; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ I commented...

For example, you want to count 2 (not 4) occurrences of the pair 'aa' in text 'aaaaa', because merge() can replace it just 2 times. In other words the counted...

Debugging at funtion level

9

## **User description** This PR is based on the observation that ChatGPT is not able to do the computation necessary to calculate the output of the whole code for a...

Alfa AWUS036ACU

6

this device seems to have Chipset Realtek RTL8812BU, not mt7921aun source: https://alfa-network.eu/awus036acu

This PR reimplements RegexTokenizer._encode_chunk() using dynamic programming to return a guaranteed optimal (minimum number of tokens) tokenization of a chunk. After training, the vocabulary is fixed. During encoding, _encode_chunk() is...

Deduplication of text chunks with frequency count, training and encoding 5x speedup

2

In `RegexTokenizer`, the training text is initially split into chunks, and further processing is performed on individual chunks. This PR optimizes the process by retaining only unique chunks and their...

Uses Regex instead of fancy-regex - 6x speedup

3

This PR realizes the wish expressed in current code to use the faster `Regex`. The text is splitted to pieces, before tokenization, according to regular expression patterns. This PR drops...

This PR introduces a caching mechanism in `_encode_ordinary_native()`, which stores the tokens for each "piece" of text. When a piece of text is repeated, its tokens are retrieved from the...

… for beam search or best-of-n sampling. If run with batch of size 1, the current code does not raise an error, because of Pytorch's Broadcasting. But by bigger batch...