infinity

infinity copied to clipboard

Support Integration with KServe

1

### Feature request [Kserve](https://github.com/kserve/kserve) is a Kubernetes based engine for predictive and generative AI models and provides abstraction for popular model servers like Huggingface TEI (https://github.com/kserve/kserve/pull/3743), Tensorflow,PyTorch etc. Request to...

indranilr

Add a End-to-end unit test for image embeddings and audio embeddings

2

### Feature request image embeddings, and audio embeddings have currently insufficent coverage in the openai api server. ### Motivation It would be great to have a test testing this end-to-end...

michaelfeil

when use engine optimum device tensorrt，startup fail

5

### System Info infinity_emb v2 --model_id /home/xxxx/peg_onnx --served-model-name embedding --engine optimum --device tensorrt --batch-size 32 OS: linux model_base PEG nvidia-smi: cuda version 11.8, tensorrt: 8.6.1 ### Information - [ ]...

weibingo

The request to the /v1/embeddings endpoint returned a 404 status code.

2

### Model description When I loaded the embedding model and tested the request, it returned a 404 status code. Is this because Infinity does not support requests in the form...

monkdharma

prepend v1 to OpenAI compatible APIs

3

### Feature request prepend v1 to OpenAI compatible APIs ### Motivation This allows us to integrate infinity the same way as other openai compatible API engines into KubeAI: https://github.com/substratusai/kubeai PR:...

samos123

Reranker dynamic quantization

1

### Feature request --dtype support for rerankers ### Motivation Easily quantize cross encoder models ### Your contribution . (will look into this)

rawsh-rubrik

Issue running cross-encoder onnx model exported with optimum-cli

2

### System Info py3.10 infinity-emb 0.0.55 Running with optimum engine fails: ``` INFO 2024-09-13 15:17:02,874 datasets INFO: PyTorch version 2.4.0 available. config.py:59 INFO: Started server process [76741] INFO: Waiting for...

rawsh

jinaai/jina-reranker-v1-*-en does not work with optimum

1

### System Info py3.10 infinity-emb 0.0.55 ``` INFO 2024-09-13 15:19:59,927 datasets INFO: PyTorch version 2.4.0 available. config.py:59 INFO: Started server process [76898] INFO: Waiting for application startup. INFO 2024-09-13 15:20:01,042...

rawsh

Write a custom flash-attention function for the deberta model.

1

### Model description I used `michaelf34/infinity:0.0.55` to deploy mixed_bread_large reranker. The container is up and I am well capable of pinging the model using python requests, but it is a...

wolfassi123

Question: Support for colbertv2.0 ?

8

### Model description Hi dear: Thanks for your source code. can support for colbertv2.0 deployment ? Thank you! ### Open source status - [ ] The model implementation is available...

shatealaboxiaowang