datasketch
datasketch copied to clipboard
Proposal: DDB storage layer
Hi,
It would be nice to have dynamoDB as a storage layer. I have implemented a basic version of this within storage.py
. Please could someone take a look to see if this is something the project would be willing to incorporate - and if i have a sensible approach in implementation. I dont believe this is "production ready" quality code yet, but i think . early feedback is useful!
This is interesting. Why Dynamo DB?
Its an awful lot cheaper than the redis instances we're using at the moment - and there's practically zero management overhead to boot!
A few things to consider:
- LSH's
query
method probes multiple hash tables, is this going to be efficient for DynamoDB (think network latency) comparing to an in-memory Redis instance on the same node as the LSH? - What is the memory footprint of your index and why not use
pickle
package to save the index to the disk periodically?
- Yeah its not overly performant - overall requests take about 500 millisec. However this is "good enough" for our use-case
- Concurrency mostly - We're going to be running this within AWS Lambda functions so there is no shared filesystem. Seems easiest to move the state out of application layer into the database layer
I see. This is quite interesting. Please give us some time.
Very interesting. @edholland can you recommend a ddb emulator to help with the dev setup, and CI?