mlx_lm.batch

Open psm-2 opened this issue 10 months ago • 2 comments

Can you add the functionality of MLX batches similar to ParaLLM?

https://github.com/willccbb/mlx_parallm/issues/9#issuecomment-2646443736

Feb 09 '25 19:02 psm-2

There is an open PR to add batch support for mlx-lm: https://github.com/ml-explore/mlx-examples/pull/948

Will try and get it landed soon if possible.

Feb 10 '25 14:02 awni

There is an open PR to add batch support for mlx-lm: #948

Will try and get it landed soon if possible.

Really need it, we definitely will face concurrent calling from users.

Feb 17 '25 04:02 qinxuye