singularity

Is there any reason to not make this app available on ARM Macs? If not, please enable the option on App Store Connect as it will enable Mac users to...

How to benchmark for speedup and acceptance rate?

7

Sorry for asking a possibly obvious question but it would be better if the documentation makes this clear.

Support automatically calculate max_total_token_num

7

In [ApiServerArgs.md](https://github.com/ModelTC/lightllm/blob/main/docs/ApiServerArgs.md), an algorithm was introduced to calculate the optimal `max_total_token_num` argument. This process can be automated, and this PR introduces this feature. The `max_total_token_num` argument now defaults to None....

Support Vision Language Models

1

Recently, multimodal models are gaining traction. It would be better if this project supported them. The underlying llama.cpp already has support for [vision language models](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#multimodal). So this shouldn't be too...