pinecone-datasets icon indicating copy to clipboard operation
pinecone-datasets copied to clipboard

to_index() should always use gRPC for bulk upserts

Open igiloh-pinecone opened this issue 2 years ago • 0 comments

@miararoy my bad, I missed this in #22. That this code should never have been merged - it breaks one of the key principles behind pinecone-datasets

Problem

One of the design principles of pinecone-dataset from day one was providing fast bulk upserts via gRPC. which isn't optional. The only change from version 0.5 to 0.6 should have been the underlying client - from still beta Client 3.0 to Client 2.2[grpc].
pinecone-datasets is not meant to support REST based upserts, which can be achieved through the client directly.

Solution

Made pinecone-client[grpc] a mandatory requirement, and use GRPCIndex as the only supported index type

Type of Change

  • [X] Bug fix (non-breaking change which fixes an issue)
  • [ ] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [ ] This change requires a documentation update
  • [ ] Infrastructure change (CI configs, etc)
  • [ ] Non-code change (docs, etc)
  • [ ] None of the above: (explain here)

Test Plan

Full coverage in current unit tests

igiloh-pinecone avatar Aug 15 '23 18:08 igiloh-pinecone