rjmehta1993

flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi

13

installing all from scratch. torch: 2.3.0 flash-attn: 2.5.7 exllama: 0.0.19 Still getting the error: flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi

Exllamav2 inference with EXL Quants

1

Do you support Exllamav2 backend for the inference that supports exl quants? The current alternative is vllm but that doesn't support EXL quants. Also, after running a perplexity test, EXL...

Remove Conda from Docker Installation

1

### System Info The docker file has non-commercial conda installation whereas the tgi is Apache-2.0 License. Installing tgi through docker creates conda license violation. https://github.com/huggingface/text-generation-inference/blob/main/Dockerfile ### Information - [x] Docker...

### Feature request TGI should read the config.json and apply the rope scaling and factor from the config.json parameter. ### Motivation Many inference engines auto-apply the rope scaling and rope...

Buy works fine. but when i sell, below is the error stock_symbol = 'NVDA' rh.order_sell_market(symbol=stock_symbol, quantity=1) OUTPUT: -> rh.order_sell_market(symbol=stock_symbol, quantity=1) -> Out[5]: {'non_field_errors': ['Invalid Good Til Canceled order.']}