Jacek

> Since the original models are using FP16 and llama.cpp quantizes to 4-bit, the memory requirements are around 4 times smaller than the original: > > * 7B => ~4...

> > 32gb is probably a little too optimistic > > Yeah, 38.5 GB is more realistic. > > See https://github.com/ggerganov/llama.cpp#memorydisk-requirements for current values I see. That makes more sense...

Agreed. I'm planning to write some wrappers to port llama.cpp using the new llama.h to other languages and a documentation would be helpful. I am happy to look into writing...

> @SpeedyCraftah go for it, here is a rough overview: > > ```c++ > const std::string prompt = " This is the story of a man named "; > llama_context*...

> Thank you for the instruction. It will be super helpful to have a minimal example of how to fire up the API, and import it from Python as a...

@Green-Sky If you don't mind me asking, how do I go about increasing the batch size of the prompt? I tried something naive but it just seems to be resulting...

> @SpeedyCraftah any update on this? Going well! I am finished with the final mock-up, now just needs some polishing, size_t conversion warning fixes and comments, then it's ready to...

Instead of shutting down the bot everytime

FYI guys I have made a type definition for this package in the definitely typed repo since the author looks like they abandoned the package. You should now be able...

Is this going to be considered? I am running into performance issues with this package on Ubuntu running node 16.