feat(compatibility): Add compatibility with Weaviate client v4.

[Fiwex]

Maintain backward compatibility with v3 client. Fix #787

[CShorten]

Thank you so much, I can take a look this evening!

[CShorten]

Hey sorry for the delay, maybe @hsm207 would have a second as well. I will hopefully be able to look at this tomorrow morning EST.

[hsm207]

Why do we want to maintain compatibility with v3 client API?

[Fiwex]

Thanks for the replies!

I decided to maintain backward compatibility with the v3 client API in this PR for smooth transition for users upgrading to the latest version.

Alternatively, I could keep v3 support for now but add deprecation warnings to encourage users to upgrade to v4. Or if you prefer, only keep the v4 compatibility.

Let me know your thoughts. I'm happy to modify the PR based on DSPy priorities.

[hsm207]

I'm not a member of the DSPy team (I work at Weaviate). My thoughts are:

We recommend users use the v4 API whenever possible. Given that this integration is pretty recent and minor, I don't think it will cause users significant disruption to upgrade.

[Fiwex]

@CShorten Just tel me what you prefer and I'll change the PR accordingly.

[CShorten]

Thanks @Fiwex I think this is solely a positive change. Thank you for the thoughts @hsm207, I agree with the sentiment, but we are indeed seeing some issues arising and I can't see any drawbacks of this approach.

[arnavsinghvi11]

Hi @Fiwex @CShorten , just catching up on old PRs - is this ready to merge with the current approach?

[CShorten]

Hey @arnavsinghvi11! No I don't believe so, it looks like the PR is removing default arguments from the hybrid search -- which I'm not 100% sure the utility of, cc @hsm207 @tsmith023.

I will try to take a look further as soon I can!

[betimd]

It would be awesome if this PR is merged ASAP. I'm having a ton of issues on my solutions, especially when I have to integrate other libs, that are specialized on specific tasks and they are depended on current version of package. It's conflicting with 3.* version hugely. Ex. I'm using DSPy and LlamaIndex import data functionality, while latest use updated dep package, because of former one I have to downgrade and have a ton of errors.

[CShorten]

Hey @betimd! Thank you so much for telling us more about this problem, really happy to hear about use cases with DSPy, LlamaIndex, and Weaviate! I have reviewed the PR and will merge it to add support for using DSPy with Weaviate's v3 and v4 Python clients.

P.S. @hsm207, arguments like hybrid_alpha and hybrid_fusion_type are supported with **kwargs. I really like this approach because of how we can add reranking, filters, and things like this.