cube
cube copied to clipboard
cube-js
Support Dynamodb in addition to Redis as a Cache server
Is your feature request related to a problem? Please describe. Currently only redis is supported as the cache server. Dynamodb is available at a much cheaper price than redis and would help to reduce the overall cost of the deployment.
Describe the solution you'd like Support Dynamodb as additional cache server.
@gauravlanjekar Thanks for posting this! I think it makes sense to support it. We need to check one thing though: if DynamoDB is capable of providing serialized read/write capabilities. Redis is used also to store query queue which requires this sort of capabilities.
By serialized do you mean the storing and serializing arrays and objects? Dynamodb does support this.
If you mean the ordering of the items it might be possible to do something with sort keys to look up items in an order. Is there any documentation on how the queue should work?
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_AttributeValue.html
I would be happy to take a look at this.
Would the right place to start be to create DynamoDB queue driver with the same interface as https://github.com/cube-js/cube.js/blob/7b62c1d8c8e5ee74118eac467bacffb83c6fcb3e/packages/cubejs-query-orchestrator/orchestrator/RedisQueueDriver.js#L5
@Ankcorn Hey Thomas! I mean serializable transaction isolation. Basically we need to ensure https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transactions.html is suitable to implement Cube.js query queue driver.
What purpose does the transaction serve? I see there are the blocking get results methods in the redis driver. Strugling to visualise exactly what this is doing because im not that familiar with redis's methods beond a simple cache use case.
The key structure of the current solution makes me think that it would map well to a single dynamodb table with few changes.
The bits I think I would strungle to implement in the same way are where operations like rpush and lpop are used where dynamodb does not have an equivelent. If there was some documentation describing what they did then I am sure we could come up with a data structure in dynamodb that achieves the same results
@Ankcorn So Redis is used to sync queue and deliver results. It's used as a mutually exclusive lock implementation as well as a way to push results delivery instantly as those are available instead of polling for them.
@Ankcorn I'm curious what are main points of using DynamoDB instead of Redis besides pricing? Maybe you want to implement it for cache and leave queue in Redis?
Appologies for the radio silence
@Ankcorn So Redis is used to sync queue and deliver results. It's used as a mutually exclusive lock implementation as well as a way to push results delivery instantly as those are available instead of polling for them.
Sounds like dynamodb wouldn't be able to fulfill the role redis does that effectively in terms of the queue. There are some options like dynamodb streams to push the results on completion, and other things but would never be as fast and flexible as redis.
@Ankcorn I'm curious what are main points of using DynamoDB instead of Redis besides pricing? Maybe you want to implement it for cache and leave queue in Redis?
We would like to cache certain queries for ever. Queries like top y of december 2019 for customer x
@Ankcorn I see. For durable cache you should use https://cube.dev/docs/pre-aggregations#top instead. Redis cache is to handle database bursts basically.
That makes total sense, I will do some research and give it a go!
Thank you
Finally, no one is working on itοΌFor me, I just want a pure serverless solution, but the redis seems the only exception for now. I think I have enough experience with DynamoDB but just a new bee for cube.js. Maybe I can help to work on it if it's a reasonable idea.
π a quick reminder that we will be replacing Redis with Cube Store as announced in this blog post.
I believe that this issue is not relevant anymore since Cube Store has replaced Redis for query queue and cache management.
Docs: https://cube.dev/docs/product/deployment#redis
Announcement: https://cube.dev/blog/how-you-win-by-using-cube-store-part-1
