integration-guides icon indicating copy to clipboard operation
integration-guides copied to clipboard
verified publisher icon meilisearch

Changes related to Meilisearch v1.13.0

Open curquiza opened this issue 11 months ago • 6 comments

This issue gathers the changes related to the v1.13.0 of Meilisearch that will impact the integrations scope.

📅 Release date: 17th February

Timelines & steps

Pre-release

  • [x] With the help of the changelog & this CI, define which integrations should be updated and how (New feature? Update README? Update tests?)
    • [x] Fill in the "What to implement?" section below in this issue 👇. Minial implementation: PHP, JS, meilisearch-js-plugins.
    • [ ] Open implementation issues in repositories that need implementation if necessary for developers who will work on the tasks
    • [x] Discuss with the Product team if needed. At least share this issue to let them know about the decisions.
  • [x] Create a branch by running Octopus script: only open branches for the integrations we choose to update (defined in the previous step) + Kubernetes repository + Cloud provider repository (changing the version)
  • [x] Update integrations according to the decisions (cf "What to implement?" section below in this issue 👇) ⚠️ If possible, this step is done before pre-release, once the feature is ready thanks to the prototype released by the engine team
    • [x] JS
    • [x] PHP
    • [x] Dart
    • [x] Go
    • [x] Java
    • [x] Python
    • [x] Ruby
    • [x] Rust
  • [x] Add code samples for the chosen up-to-date integrations with the new version of Meilisearch
  • [x] Update the library version of the related integrations and prepare the changelogs

Release day

  • [ ] Release the integrations or only merge the related PRs (sometimes a release is not needed)
    • [x] JS v0.49
    • [x] PHP v1.13
    • [x] meilisearch-js-plugins (at least update meilisearch-js version)
    • [ ] Dart v0.17
    • [x] Go v0.31
    • [x] Python v0.34
    • [x] Rust v0.28
  • [x] Merge the related PR in K8s repository
  • [x] Publish DevOps tools:
    • [x] create the git tag
    • [x] publish images (steps are in CONTRIBUTING.md)
  • [ ] Remove useless AWS images (by using our internal script)
  • [x] Open issues in the repositories that are not up-to-date with the latest version of Meilisearch (including code samples)

Post-release

  • [X] Open issues post-release for stabilization of AI
  • [ ] Update the following SDKs with the missing capabilities for AI search
    • [ ] Java: https://github.com/meilisearch/meilisearch-java/issues/817
    • [ ] Python: https://github.com/meilisearch/meilisearch-python/issues/1081
    • [X] Ruby: https://github.com/meilisearch/meilisearch-ruby/issues/608
  • [ ] Update JS and PHP with
    • [ ] Remote federated search
    • [ ] usedDatabaseSize (stats)
    • [ ] Document Database (stats)
    • [ ] Embedding database (stats)

What to implement?

AI stabilization

Related issue in the engine: https://github.com/meilisearch/meilisearch/issues/4733

Usage: https://meilisearch.notion.site/v1-13-AI-search-changes-17a4b06b651f80538b65d31724545def

  • removal of the vectorStore experimental feature
  • new error codes

TODO:

Postponed to v1.14.0 ### Granular filterable attributes

Related issue in the engine: https://github.com/meilisearch/meilisearch/issues/5163

Usage: https://meilisearch.notion.site/API-usage-Settings-to-opt-out-indexing-features-filterableAttributes-1764b06b651f80aba8bdf359b2df3ca8?pvs=74

  • accept the new formal of filterable attributes (the old one is still allowed)

TODO:

  • [ ] JS: adapt code base + fix tests
  • [ ] PHP: adapt code base + fix tests

Post release day - implement missing feature for vector store

Spec

  • [ ] embedders setting. Methods getEmbedders, updateEmbedders, resetEmbedders. Also, the method updateSettings should be able to accept the new embedders field. Here is the list of the acceptable sub fields:
    • [ ] source sub field is available and accepts: ollama, rest, openAI, huggingFace and userProvided
    • [ ] apiKey sub field is available (string) - optional because not compatible with all sources. Only for openAi, ollama, rest.
    • [ ] model sub field is available (string) - optional because not compatible with all sources. Only for ollama, openAI, huggingFace
    • [ ] documentTemplate sub field is available (string) - optional
    • [ ] dimensions - optional because not compatible with all sources. Only for openAi, huggingFace, ollama, and rest
    • [ ] distribution - optional
    • [ ] request - mandatory only if using rest embedder
    • [ ] response - mandatory only if using rest embedder
    • [ ] documentTemplateMaxBytes - optional
    • [ ] revision - optional, only for huggingFace
    • [ ] headers - optional, only for rest
    • [ ] binaryQuantized - optional
  • [ ] Search
    • [ ] hybrid search parameter, with sub fields semanticRatio and embedder. embedder is mandatory if hybrid is set.
    • [ ] vector parameter is available
    • [ ] retrieveVectors parameter available
    • [ ] semanticHitCount in search response
    • [ ] Accept _semanticScore in the search response (optional)
    • [ ] vector should be returned in the search response, but optional (because depends on search parameters)
    • [ ] _vectors should NOT be present in the search response
  • [ ] Similar. Implement searchSimilarDocuments associated with the POST /indexes/:uid/similar. Do NOT implement with GET.

TODO

  • [ ] Java
  • [ ] Python
  • [X] Ruby

Post-release: Remote federated search requests (exp)

Related issue in the engine: https://github.com/meilisearch/meilisearch/issues/4980

Usage: https://meilisearch.notion.site/API-usage-Remote-search-request-f64fae093abf409e9434c9b9c8fab6f3

  • New remote route
  • accept remote search parameter in /multi-search route

TODO:

  • [ ] JS: adapt code base + fix tests
  • [ ] PHP: adapt code base + fix tests

Post-release: New usedDatabaseSize in /stats

Usage: https://github.com/meilisearch/meilisearch/issues/5318

  • [ ] Accept the usedDatabaseSize in the response of the getStats method

TODO

  • [ ] JS
  • [X] PHP: no need to change 🎉

Issue to open after the release: .NET, dart, go, Java, Rust, Swift (not Ruby, or Python)

Post-release: Add embbedings database metrics to stats route

Usage: https://github.com/meilisearch/meilisearch/issues/5321

  • [ ] Accept indexes.INDEXNAME.numberOfEmbeddedDocuments in the response of the getStats method
  • [ ] Accept indexes.INDEXNAME.numberOfEmbeddings in the response of the getStats method

TODO

  • [ ] JS
  • [X] PHP: no need to change 🎉

Post-release (v1.13.1): Add documents database metrics to stats route

https://github.com/meilisearch/meilisearch/issues/5319

  • [ ] Accept indexes.INDEXNAME.rawDocumentDbSize in the response of the getStats method
  • [ ] Accept indexes.INDEXNAME.maxDocumentSize in the response of the getStats method
  • [ ] Accept indexes.INDEXNAME.avgDocumentSize in the response of the getStats method

TODO

  • [ ] JS
  • [X] PHP: no need to change 🎉

curquiza avatar Jan 22 '25 09:01 curquiza

Granular filterable attributes Related issue in the engine: https://github.com/meilisearch/meilisearch/issues/5163 Usage: https://meilisearch.notion.site/Opt-out-settings-v2-usage-15c4b06b651f8188ae57fca7aa73b527

accept the new formal of filterable attributes (the old one is still allowed)

@curquiza I don't think that notion link is correct.

ellnix avatar Jan 22 '25 11:01 ellnix

Thank you, I changed

curquiza avatar Jan 22 '25 12:01 curquiza

@Strift, the Granular filterable attributes feature will not be ready for v1.13.0, but will be done in v1.14.0 Just got the new from engine team, who had difficulties managing the flood of bugs and the sprint lately

curquiza avatar Jan 23 '25 14:01 curquiza

How do we want to go about the rust integration? (I have not updated my PR to remove the experimental part, but can do so)

Kontext:

  • https://github.com/meilisearch/meilisearch-rust/pull/554

These are the options

  • [x] remove ExperimentalFeatures::set_vector_store
  • [ ] mark ExperimentalFeatures::set_vector_store as deprecated and change to a no-op

And about reviewability:

  • [ ] include in the same PR
  • [x] make a new PR

CommanderStorm avatar Jan 24 '25 19:01 CommanderStorm

There seems to be precedent for this @CommanderStorm

https://github.com/meilisearch/meilisearch-rust/commit/d12b21ba243afaad6ff4377a9f4aa0551dad787c

https://github.com/meilisearch/meilisearch-rust/pull/544

Looks like it was just removed (I suppose it was "experimental" after all). You know better though, of course doing something different could be appropriate in your case. Just thought I'd share.

ellnix avatar Jan 24 '25 20:01 ellnix

Hey there, I opened the issues to track the missing 1.13 features in the relevant SDKs and closed the duplicate issues related to previous iterations of AI-powered search.

Strift avatar Feb 20 '25 04:02 Strift

I guess we can close this issue @Strift, no?

brunoocasali avatar May 15 '25 15:05 brunoocasali

We still have to merge https://github.com/meilisearch/meilisearch-java/pull/827, and then we'll be done 🥲

Strift avatar May 21 '25 06:05 Strift