lorax
lorax copied to clipboard
Can I deploy the service using Lorax without using lorax-launcher to start?
Can I deploy the service using Lorax without using lorax-launcher to start, and instead load the model in the code?
Similar to HF and VLLM, I can use the following code to load the model.
# vllm
from vllm.engine.async_llm_engine import AsyncLLMEngine
model = AsyncLLMEngine.from_engine_args(engine_args)
# hf
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"model_id",
torch_dtype="auto",
device_map="auto"
)