David Hinkle comments

Repositories
Issues
Comments

Results 5 comments of


                                            David Hinkle

Missing license

Would love to see the License issue fixed. It makes it a lot easier for those of us in the commercial space to engage.

[Feature Request] Option to specify multiple available miner quality types

How do you select the miner type and ore quality?

Batched Inference to Improve GPU Utilisation

I feel like this is very important, if they don't implement batch inferencing I can't really consider it over llama.cpp's GBNF grammers.

Batched Inference to Improve GPU Utilisation

I'm not very strong on the theory, but llama.cpp does support continuous batch inference with a grammar file. It had grammar support and continuous batching support for a while, but...

Question about batched inference and async_mode

I need to understand this as well if it is possible.