icyboy™

Route53 not support Failover

1

aws route53 supports failover, but not work in coredns route53 plugin route53 failover config: ``` s-search-hdfs-nn.test.release | A | Failover | Secondary | 10.60.8.187 10.60.0.129 10.60.11.82 s-search-hdfs-nn.test.release | A |...

Request failed during generation: Server error: 'FlashMixtral' object has no attribute 'compiled_model'

3

### System Info Request failed during generation: Server error: 'FlashMixtral' object has no attribute 'compiled_model' server/text_generation_server/models/flash_mistral.py 516 ### Information - [ ] Docker - [ ] The CLI directly ###...

### Feature request https://github.com/FasterDecoding/SnapKV ### Motivation SnapKV: Cache compression technique for faster LLM generation with less compute and memory In a recent paper, authors introduced 𝗦𝗻𝗮𝗽𝗞𝗩 as a novel technique...

DistServe support

1

### Feature request https://github.com/LLMServe/DistServe ### Motivation DistServe improves the performance of large language models (LLMs) serving by disaggregating the prefill and decoding computation. Existing LLM serving systems colocate the two...

AttributeError: 'MixtralLayer' object has no attribute 'mlp'

3

### System Info 2024-06-26T08:59:14.473641Z ERROR text_generation_launcher: Error when initializing model Traceback (most recent call last): File "/opt/conda/bin/text-generation-server", line 8, in sys.exit(app()) File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 311, in __call__ return get_command(self)(*args, **kwargs)...

fp8 weight load failed IndexError: list index out of range

2

### System Info ``` 2024-08-13T06:17:44.049654Z ERROR shard-manager: text_generation_launcher: Shard complete standard error output: 2024-08-13 06:17:41.545 | INFO | text_generation_server.utils.import_utils::75 - Detected system cuda /opt/conda/lib/python3.10/site-packages/text_generation_server/utils/sgmv.py:18: UserWarning: Could not import SGMV kernel...