Martin Evans

I wouldn't expect a batch-decoding executor any time soon, it's going to be a _lot_ of work to design the API to support all of the various features in an...

LLamaSharp actually should have pretty good Chinese support. @AsakusaRinne has done a lot of work with encoding for Chinese models, and I did some work on detokenisation for complex characters...

@Xsanf could you open an issue with the errors you're seeing? Unity isn't one of our test targets, but if there are small tweaks we can make to improve compatibility...

The `BatchedExecutor` is something I've been developing for a few months, it's intended as a new "low-ish level" foundation which future higher level executors can be built on. It's the...

Could you try running this: ```csharp var model = LLamaWeights.LoadFromFile("your_model_path"); Console.WriteLine(model.NewlineToken); ``` The code that's crashing is this: ```csharp var nl_token = model.NewlineToken; var nl_logit = logits[(int)nl_token]; ``` So it...

I'm not certain, but it doesn't seem correct for any model to be returning `-1` for the newline token. That would mean the model has no concept of newlines, which...

If multiple models are showing the same thing I guess that must be normal. Very weird! In that case I think the `NewlineToken` method should be updated to return `LLamaToken?`...

In the #662 PR I've modified how tokens are returned from the LLamaSharp API so it returns nullable tokens and fixed all of the call sites to handle this. I...

Note that any tests added for this probably need to be disabled on CI, the runner don't have the memory or the compute to handle long contexts!

What model file are you trying to use?