Nicolas Patry comments

Results 977 comments of


                                            Nicolas Patry

Model warmup fails after adding Triton indexing kernels

can also every confirm they are using `3.0.1` and not `latest` ?

Model warmup fails after adding Triton indexing kernels

> Currently, I’m forced to assing GPUs to container manually and not through the Nvidia container toolkit, Can you elaborate ? It might be a potential culprit.

Model warmup fails after adding Triton indexing kernels

It's starting to look like a podman bug... I cannot reproduce with the minimal reproducer.... Which version of podman are you using ? (`5.3.1` here)

Model warmup fails after adding Triton indexing kernels

I can't reproduce any of the issues even with podman on my end... What are the host configs? GPU, cuda version, every potential service, driver on the nodes etc.. ?...

feat: add cli for tokenizer and training

If it's that small, and adds new dependencies, I feel like it should be it's own crate. tokenizers is a library, it shouldn't be a CLI as well.

Unused Unicode Character Filter

Thanks for the great PR !! You created 3 booleans. Shouldn't we maybe use a flag array so users can filter in/out any particular categories maybe ? Are there no...

Faster Whitespace PreTokenizer (Drop-in Replacement)

Have you verified exhaustively that the split is EXACTLY the same on all sort of utf-8 boundaries ? Utf-8 is complex enough that I don't feel a 20% speedup on...