ktransformers icon indicating copy to clipboard operation
ktransformers copied to clipboard
Published 3 weeks ago • verified publisher icon kvcache-ai

AMX only? re: KTransformers+SGLang Inference Deployment

Open fernandaspets opened this issue 1 month ago • 3 comments

Hi is the KTransformers+SGLang Inference Deployment only supported for AMX CPUs? Or can we try this with Epyc as well?

fernandaspets avatar Nov 15 '25 05:11 fernandaspets

Yes, we can try AMD's CPU with AVX512 support right now. And the see PR:#1600 will enhance the performance for AMD int8 (but haven't finished yet, you can have a try if you want). And we also support AMD with the llamafile implementation. see the Q&A's link for the kt-kernel README #1608

KMSorSMS avatar Nov 15 '25 10:11 KMSorSMS

Do Zen4 cpus of avx2 have any chance?

jli113 avatar Nov 18 '25 03:11 jli113

Haven't tested. After the PR is merged, you can give it a try to see the performance for int8 prefill.

KMSorSMS avatar Nov 18 '25 03:11 KMSorSMS