HuggingFace Tokenizers as Collate Functions Timing 🤗 🤖 | DeepSchool

[utterances-bot]

HuggingFace Tokenizers as Collate Functions Timing 🤗 🤖 | DeepSchool

Timing comparison of tokenizer as collate function and after batching

https://sachinruk.github.io/blog/pytorch/huggingface/2021/11/17/collate-fn-vs-normal.html

[thesofakillers]

Hey, it seems that the utterances integration is broken on your website so i'm directly commenting here.

Thanks for the blogpost. It is an interesting result. Have you since fully switched to collation outside of the dataloader? Or is there something keeping you to the traditional way of passing collate_fn to the dataloader?