DiceDB
Add support for in-memory key/value based ETL
Is your feature request related to a problem? Please describe. What if I want to find MAX/SUM/%ile of filtered keys/values? Doing this today is a big pain as it requires a massive compute power.
Given that Dice DB is built on Golang and it has super-efficient Goroutines, what it we build capability of doing in-memory ETL through simple Piped Commands?
Describe the solution you'd like Build an ability to represent and accept ETL command execute them on multiple cores on mutually exclusive subset of data and quickly compute the response.
This way, Dice DB is single threaded for execution, but each command execution can be multi-threaded leveraging all the cores of the underlying hardware.
Describe alternatives you've considered There is no way to do value based operations and key based filters are expensive. No matter what we are not leveraging all the underlying cores
Additional context This would open up the whole world of multi-core data structures and algorithms to be added to Dice.
@arpitbbhayani Would like to pick up this feature. Are you chasing any timeline for this feature? As I am new to DiceDB and GoLang, might take a little longer.
@himanishm07 the scope of this is still blurr. I have added it so that I don't miss it.