feast icon indicating copy to clipboard operation
feast copied to clipboard

Bigtable online store support

Open adchia opened this issue 4 years ago • 0 comments

Is your feature request related to a problem? Please describe. Existing online store options for GCP (i.e. MemoryStore/Redis or Datastore) make trade-offs between performance and simplicity. Many users come to Feast with existing usage of Bigtable and want to use that. Other users want lower latency retrieval (Datastore tends to be significantly slower) or fast range scans.

Describe the solution you'd like A Go-based online connector for Bigtable, which we then can call into from the Python feature server via a Go subprocess

  • [ ] Mini-RFC on how range scans over online stores might fit into Feast
  • [ ] Build basic Go online store connector for Bigtable
  • [ ] Call into Go online store connector via a Go subprocess (in progress)
  • [ ] Optimize Bigtable connector for batch retrieval (i.e. group row keys on the client side before sending since Cloud Bigtable does not work as well if the batch spans multiple nodes)

Describe alternatives you've considered Using the Python Bigtable client. The Bigtable team told us their Go client is significantly more mature.

adchia avatar Feb 18 '22 21:02 adchia