Nicolas Patry
Nicolas Patry
Can you open a new issue please ? It's not really good practice to resurrect old threads as it pollutes searches with potentially irrelevant content, and makes your issue which...
Ok looked at this issue (I will copy it into a new issue once there's one). The error is because of the current tokenizer format which expects the `merges` part...
Hi @markhughes I think given the age of neon, it might even be more practical to start over from scratch. Maybe take the current version as a starting point but...
I'm not sure anything else than rebasing an checking everything works is needed actually.
> here's hoping this gets merged sooner rather than later I don' t like merging old stuff without proper checking. I think training something on `main` on `big.txt` then the...
Thanks so much for looking into it !!
Very old issue that I somehow missed. sorry about that. For future readers, nvidia drivers tend to prefer matrices of shapes of size multiple of eights including the sequence_length that...
Totally understandable. What kind of pre-tokenizer are you saving ? If some building blocks are missing we could add them to make the thing more composable/portable/shareable.
No. custom is python code, it's not serializable by nature.
PRs are welcome. But rewriting the neon bindings is just too big an endeavor for us at the moment. `unstable_wasm` feature and example could be an escape hatch. (It's unstable...