Michał Moskal

I suggest you try adafruit forums. Message ID: ***@***.***>

Happy to chat @AaronFriel here or otherwise!

samples could be copied from pyctrl readme and translated

It looks like fork() on Windows is going to be a major problem https://stackoverflow.com/questions/985281/what-is-the-closest-thing-windows-has-to-fork A solution to that would be to create a separate process per request/sequence group and have...

could you elaborate?

closing for lack of clarity

Yes, that would be nice! PRs are welcome!

also https://github.com/microsoft/aici/commit/97227956622a82f08e379514278bc3597db16a46 https://github.com/microsoft/aici/commit/6390cbe4149b8a992a7e0df80ba9aebaebd9bebb

This requires upgrading version of llama.cpp used. I should get to this sometimes this week or next.

It looks like it's only a problem with the llama.cpp backend (when using orca deployment it seems deterministic). Need to look into llama.cpp sampling I guess...