mlc-llm

mlc-llm copied to clipboard

## 🚀 Feature Similar to a standard request to the API, the final stream chunk should have a usage field to give prompt/generated token counts. This can be sent with...

TNT3530

[Bug] Still Experiencing 'Error: Using LLVM 19.1.3 with `-mcpu=apple-latest` is not valid in `-mtriple=arm64-apple-macos`, using default `-mcpu=generic`'

2

## 🐛 Bug ## To Reproduce Steps to reproduce the behavior: 1. Build from source using MLC's TVM-Relax & their MLC-LLM Github Repos. I used a few custom options if...

BuildBackBuehler

## 🚀 Feature Embeddings support for iOS. Is this part of the roadmap? ## Motivation Ability to generate embeddings in iOS applications. ## Additional context Looking at the code I...

jondeandres

[Feature Request] Add vision model flag to model record

1

## 🚀 Feature Add a field to the `ModelRecord` to indicate whether a model is a vision model like `Phi-vision` models. ``` data class ModelRecord( @SerializedName("model_url") val modelUrl: String, @SerializedName("model_id")...

Neet-Nestor

[Bug] Running Quick Start Example in Windows gives Error: 'InternalError: Check failed: (it != n->end()) is false: cannot find the corresponding key in the Map and MLCEngine' object has no attribute '_ffi'

8

## 🐛 Bug MLCengine code in quickstart guide on CPU fails with > 'InternalError: Check failed: (it != n->end()) is false: cannot find the corresponding key in the Map' followed...

Seoulsim

This PR adds the support for vllm and llama.cpp backend, especially for json generation.

cyx-6

I am trying to use the direct output of `MLCEngine` class but I do not have any clue how to get the output based on the MLC-LLM docs. Is there...

pjyi2147

[Bug] large concurrency service broken

4

## 🐛 Bug The service started based on Meta-Llama-3.1-70B-Instruct fp8 will crash when running a large concurrency. ## To Reproduce ### convert model refer this issue: #2982 ### start service...

fan-niu