teis-e
teis-e
### System Info - x86_64 - RTX4090 - Intel Xeon 2696v4 - 128 DDR4 2400mhz ### Who can help? @Tracin @juney-nvidia ### Information - [ ] The official example scripts...
### Bug Description ``` query_engine = index.as_query_engine( filters=MetadataFilters( filters=[ MetadataFilter( key="color", value="red", ), MetadataFilter( key="alpha", value="1", ), ], condition=FilterCondition.OR, ) ``` Only when i use either of the filters it...
### Bug Description I just build an engine with TensorRT-LLM version: 0.8.0 and get this error running TensorLLM object: `[03/27/2024-11:09:02] [TRT] [E] 6: The engine plan file is generated on...
### Bug Description Running TensorLLM object: ``` from llama_index.llms.nvidia_tensorrt import LocalTensorRTLLM llm = LocalTensorRTLLM( model_path="./model", engine_name="rank0.engine", tokenizer_dir="/root/.cache/huggingface/hub/models--Qwen--Qwen1.5-72B-Chat-GPTQ-Int4/snapshots/> completion_to_prompt=completion_to_prompt, verbose=True, max_new_tokens=640, temperature=0 ) ``` Results in: ``` [TensorRT-LLM] TensorRT-LLM version: 0.8.0Traceback...
### Bug Description I got all the necessary files provided in this [Nvidia TensorRT-LLM documantation](https://docs.llamaindex.ai/en/stable/examples/llm/nvidia_tensorrt.html) Like this in my dir: app.py model/ ├── .gitkeep ├── config.json ├── generation_config.json ├── LICENSE.txt...
I saw some other people have this issue. Is this something you have to deal with or is this a bug on my install? I can't use RTX now
Trying to run llama3 70B Hardware: 3x 4090 `deepspeed deep.py` deep.py: ``` # --------------------------------------- # New automatic tensor parallelism method # --------------------------------------- import os import torch import transformers from transformers...