Option to output result embeddings

[hwazni]

Hello, I would like to know if it's possible to get the image embeddings with a text query. For instance, text: an image of a cat, and the output would be all the image embeddings before using them in KNN. Thanks :)

[rom1504]

The flow is like this

1. Text
2. Text embedding
3. Knn
4. IDs that match specific items in image embedding index
5. Image embeddings or metadata (url, text)

Which embedding are you interested in ? Step 2 or 5 ?

[hwazni]

I am interested in the image embeddings of step 5.. Thanks.

[rom1504]

Ok yeah, that could totally be added. The embeddings are already available in the back there https://github.com/rom1504/clip-retrieval/blob/main/clip_retrieval/clip_back.py#L375

Would just need to return them in the API

[hwazni]

Is it possible to edit the backend results ? or print the embeddings ..

[rom1504]

You can make a PR to add the feature to return the embeddings in the API

[lizhiustc]

@rom1504 hi, I'm using clip_client.py to query the remote resources (url="https://knn5.laion.ai/knn-service", indice_name="laion5B"). And I also want to get the image embeddings with a text query. Making a PR to add the feature in clip_back is suitable for local data. But this resources is not the local but remote . So would you like to add the feature to return the embeddings for remote query ? Thanks a lot !

[rom1504]

Clip back is what is running remotely If a PR gets merged to add that feature, I'll update the instance at knn5.laion.ai

On Tue, Nov 15, 2022, 11:58 lizhiustc @.***> wrote:

@rom1504 https://github.com/rom1504 hi, I'm using clip_client.py https://github.com/rom1504/clip-retrieval/blob/00217440e3b0b994ec7919187928d5d53e13bab5/clip_retrieval/clip_client.py#L17 to query the remote resources (url="https://knn5.laion.ai/knn-service", indice_name="laion5B"). And I also want to get the image embeddings with a text query. Making a PR to add the feature in clip_back https://github.com/rom1504/clip-retrieval/blob/main/clip_retrieval/clip_back.py#L375 is suitable for local data. But this resources is not the local but remote . So would you like to add the feature to return the embeddings for remote query ? Thanks a lot !

— Reply to this email directly, view it on GitHub https://github.com/rom1504/clip-retrieval/issues/194#issuecomment-1315142388, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAR437TWQQFKEXPTIZ5QPFLWINUHDANCNFSM6AAAAAAREFCF3A . You are receiving this because you were mentioned.Message ID: @.***>

[lizhiustc]

Hi, I make a PR to add that feature. https://github.com/rom1504/clip-retrieval/pull/206#issue-1450024806