aihenry
aihenry
For your reference, i am running below code on my i5 PC without GPU, fast enough :) modelInUse = "codellama-13b-instruct.ggmlv3.Q4_1.bin" config = { 'max_new_tokens': 1024, 'repetition_penalty': 1.1, 'temperature': 0.1, 'top_k':...
@TheBloke 🥇 💯 👍 @marella 🥇 💯 👍 Both of you are my Hero! I learned LLM application design and integration with your LLMs and the ctransformers lib!
config = {'max_new_tokens': 2048, **'context_length': 8192, #
@marella Thank you for your hints. After re-install these, it works fine: pip install ctransformers[cuda] pip install nvidia-cublas-cu11 pip install nvidia-cuda-runtime-cu11