Bang DaeYeong

How to apply topk option each inputs?

6

Hello, first of all, thank you for creating this library. I have 2 questions. First question, I saw [this guide ](https://github.com/NVIDIA/FasterTransformer/blob/dev/v5.0_beta/docs/gptj_guide.md)and successfully started Triton Server. and Here is my request...

bug: bentoml not support gpu in mlflow?

1

### Describe the bug i am save bento with mlflow (sentence transformers) ```python def save_model_to_mlflow(self, version): signature = mlflow.models.infer_signature( self.input_data, self.output_data ) model_info: mlflow.models.model.ModelInfo = ( mlflow.sentence_transformers.log_model( model=self.model, artifact_path=self.model_name, signature=signature,...

### Describe the bug gensim word2vec model ```python import os import mlflow from gensim.models.word2vec import Word2Vec class ScappyWrapper(mlflow.pyfunc.PythonModel): def load_context(self, context): file_path = os.path.join(context.artifacts["model_path"], "scappy_base.bin") self.model = Word2Vec.load(file_path) def predict(self,...

bug: timeout configuration not work

1

### Describe the bug The timeout setting of api_server and runner is not working in bentoml. i'm using bentoml 1.0.20.post11 version The default configuration is as follows ```yaml version: 1...

why online seving slower than offline serving??

9

1. offline serving  2. online serving(fastapi)   log: INFO 12-11 21:50:36 llm_engine.py:649] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs,...