VIGHNESH SUDHAKAR comments

Repositories
Issues
Comments

Results 3 comments of


                                            VIGHNESH SUDHAKAR

can we use a gpu for increased speed and use of a bigger better llama2 model

Hi @alexng88 , I am also facing the same issues, did you get any solutions?

Issue: GPU is not used with Ctransformers even after specifying gpu_layers parameter

@dosu-bot ``` from langchain.llms import CTransformers from dotenv import find_dotenv, load_dotenv import box import yaml from accelerate import Accelerator load_dotenv(find_dotenv()) with open('config/config.yml', 'r', encoding='utf8') as ymlfile: cfg = box.Box(yaml.safe_load(ymlfile)) accelerator...

Issue: GPU is not used with Ctransformers even after specifying gpu_layers parameter

> @VIGHNESH1521 > try adding this: > > llm = CTransformers(model=cfg.MODEL_BIN_PATH, > model_type=cfg.MODEL_TYPE, > config={'max_new_tokens': cfg.MAX_NEW_TOKENS, > 'temperature': cfg.TEMPERATURE, > 'gpu_layers': 50} > ) > **llm = accelerator.prepare(llm)** > >...