lorax icon indicating copy to clipboard operation
lorax copied to clipboard

Can I deploy the service using Lorax without using lorax-launcher to start?

Open Nipi64310 opened this issue 11 months ago • 2 comments

Can I deploy the service using Lorax without using lorax-launcher to start, and instead load the model in the code?

Similar to HF and VLLM, I can use the following code to load the model.

# vllm
from vllm.engine.async_llm_engine import AsyncLLMEngine
model = AsyncLLMEngine.from_engine_args(engine_args)


# hf
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
    "model_id",
    torch_dtype="auto",
    device_map="auto"
)

Nipi64310 avatar Mar 06 '24 07:03 Nipi64310