Ben Eyal
Ben Eyal
Hello, dear maintainers! ๐ I was wondering whether a `CONTRIBUTING.md` file is planned for this repository. ๐คTokenizers is a wonderful project, and I'd love to try and contribute to it,...
Hi! Congratulations on the new release and docs! Will it be possible to bring back the search bar that was present in the previous version of the docs website? It...
Hi ๐ I couldn't find an answer, so I hope it's okay I'm asking here. I'm migrating a project to Scala 3.1.2, and have this line in my `Dependencies.scala`: ```scala...
Hi, When running `python main.py --batch_size 20 --data data/penn --dropouti 0.4 --dropouth 0.25 --seed 141 --epoch 500 --save PTB.pt` I get the following error: ``` /usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py:179: RuntimeWarning: RNN module weights...
Is there a way to enable font ligatures in fonts like Fira Code?
Hi everyone, I'm trying to decode a (faily strange) JSON encoding of SQL queries. The `Sql` case class has three points of potential recursion, all of them are `Option[Sql]`, and...
Fix Issue 15003: SentencePiece Tokenizers Not Adding Special Tokens in `convert_tokens_to_string`
Hello! ๐ This PR fixes issue #15003 and is also related to issue #11716. First, I've expanded the `test_sentencepiece_tokenize_and_convert_tokens_to_string` test in `test_tokenization_common.py` which, in addition to checking a round-trip from...
Hi! ๐ I posed this issue over at TensorFlow's GitHub, and was told I should be posting this issue here. I'm having a very weird problem which I couldn't find...
Hello, I came across a problem with the following simplified code: ```scala //> using dep "dev.zio::zio-config:4.0.0-RC16" //> using dep "dev.zio::zio-config-magnolia:4.0.0-RC16" //> using dep "dev.zio::zio-config-typesafe:4.0.0-RC16" //> using dep "dev.zio::zio:2.0.13" import zio.*...