meilisearch-rust icon indicating copy to clipboard operation
meilisearch-rust copied to clipboard

Support `embedders` setting

Open CommanderStorm opened this issue 11 months ago • 9 comments

Pull Request

Related issue

Fixes https://github.com/meilisearch/meilisearch-rust/issues/541 Fixes https://github.com/meilisearch/meilisearch-rust/issues/612

What does this PR do?

  • ~~introduces the experimental-vector-search-feature flag~~

  • Adds the required settings

    • with_embedders does use the same "API" (not using impl AsRef for items passed) as with_synonyms, as this is the closest existing
    • given set_embedders has not been implemented upstream (at least when I try to PATCH the object, it does not work)
    • only {get,reset}_embedders settings have been implemented. Said implementation goes with the work done in https://github.com/meilisearch/meilisearch-python/pull/924
  • adds the hybrid field to search via the vector search to add an end-to-end test of this feature with the huggingface configuration.

    userProvided seens more brittle, but we may want change to this instead using userProvided instead would mean (at the cost of hardcoding stuff) => lower cpu effort => no higher timeout necceeseary => aligning with meilisearch/meilisearch to only have a test for userProvided)

[!CAUTION]
This is the difficult part: I have not gotten this feature to work as I expect it. I have documented what I would expect to happen here: https://github.com/meilisearch/meilisearch-rust/blob/86334cdfa1a429eb692e3d674e4715950c47ed7e/src/search.rs#L1223-L1304

  • Where in the Meilisearch codebase could I find an e2e-test how to use feature? I searched a bunch, but could not find any relevant testcases (only userProvided)
  • Am I misunderstanding what the capabilities should be, or do I have a bad HuggingFace-embedder?

[!TIP] The specific testcase can be executed via

cargo test --package meilisearch-sdk --lib search::tests::test_hybrid  --features=experimental-vector-search -- --exact

TODO:

  • [x] find a combination of semantic search model + configuration that does not fail the assumptions (see search testcase) spectacularly

PR checklist

Please check if your PR fulfills the following requirements:

  • [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
  • [x] Have you read the contributing guidelines?
  • [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!

CommanderStorm avatar Mar 02 '24 15:03 CommanderStorm