spring-ai icon indicating copy to clipboard operation
spring-ai copied to clipboard

Published 20 hours ago • verified publisher icon spring-projects

Add support for Elasticsearch vector store

Open JM-Lab opened this issue 1 year ago • 1 comments
trafficstars

This PR adds supports for Elasticsearch vector search.

To achieve compatibility with both Elasticsearch versions 7 and 8, I implemented it using Elasticsearch's Java Low Level REST Client.

To-do:

  • [x] Add ElasticsearchAiSearchFilterExpressionConverter
  • [x] Add support for FilterExpression
  • [x] Add support for various vector functions provided by Elasticsearch

JM-Lab avatar Jan 09 '24 13:01 JM-Lab

Just finished up the Elasticsearch Vector Store support. Next, I'll be working on adding support for OpenSearch (https://opensearch.org/) vector store. Additionally, breaking down tasks for separate PR:

  • [ ] Add autoconfiguration
  • [ ] Write README

Let me know your thoughts.

JM-Lab avatar Jan 14 '24 10:01 JM-Lab

This is a great contribution. I believe @tzolov has started to review. I'll put it into 0.8.0 as a stretch goal for now.

markpollack avatar Jan 23 '24 18:01 markpollack

If we use elasticsearch-java, we might need to either split Vectorstore for Elasticsearch 7 and 8, or let users add elasticsearch-java version matching their elasticsearch server version. What's your preference? I'd love to hear your thoughts on this.

JM-Lab avatar Feb 03 '24 03:02 JM-Lab

I think moving to use the latest version that focuses on elastic search 8 and to build upon the elasticsearch-java will be the best use of everyone's time. Sorry for the delay in responding, but we are still very interested in this contribution and appreciate your interest/help.

markpollack avatar Feb 29 '24 15:02 markpollack

@markpollack I understand, and I'll update to support the latest version, Elasticsearch 8, and commit soon.

JM-Lab avatar Mar 02 '24 13:03 JM-Lab

@markpollack @tzolov Review code with Elasticsearch 8 support using elasticsearch-java library as discussed.

JM-Lab avatar Mar 03 '24 14:03 JM-Lab

Hi @JM-Lab , Finally i had some time today to start reviewing the PR. After rebasing it on top of upstream main there were some compilation issues i had to fix. Also i had to add the project the parent POM and to the BOM pom.

After this the filter expression test passes but the IT fails for half of the tests. Here is my review branch: https://github.com/tzolov/spring-ai/pull/new/gh-234-pr

Please have a look, hopefully you would be able to resolve those.

tzolov avatar Mar 15 '24 17:03 tzolov

Hi @tzolov

I've reviewed the PR. I cherry-picked your commit 'Rebase, fix compilation and style errors, add missing dependencies in parent pom and Bom pom' and resolved the issues with Elasticsearch8VectorStoreIT. Additionally, I renamed it to ElasticsearchVectorStoreIT.

Please review the changes. Looking forward to your feedback!

JM-Lab avatar Mar 17 '24 06:03 JM-Lab

Thanks @JM-Lab , 75 commits and 656 files changed, doesn't seem right. Can you please squash the commits and make sure that only the ElasticSearch related files a touched. It will be even better if you copy the ElasticSearch related files into a fresh new PR on top of the upstream main.

tzolov avatar Mar 18 '24 21:03 tzolov

@tzolov

I've just created a new PR focusing solely on the ElasticSearch files, on the main branch. You can find it here: https://github.com/spring-projects/spring-ai/pull/471.

Going to close this current one. Thanks!

JM-Lab avatar Mar 19 '24 13:03 JM-Lab

Thanks @JM-Lab I will have a look

tzolov avatar Mar 19 '24 14:03 tzolov