dice icon indicating copy to clipboard operation
dice copied to clipboard

Refactor eval to align with multi-threading implementation

Open soumya-codes opened this issue 1 year ago • 5 comments

Refactor eval to align with multi-threading design. This is a very big task and may need multiple people working together and con-ordinating to achieve this.

The tasks needs to be broken down keeping the following in mind:

  1. Break down the current eval monolith implementation into individual commands.
  2. Breakdown the individual eval commands into the following:
  • Worker level implementation: Ability to convert store layer response to client response.
  • Shard level implementation: Ability to convert the ShardRequest/StoreRequest into store operations.
  1. Ensure the refactoring supports the following scenarios properly:
  • Support commands that do not need access to store for example AUTH, PING, INFO etc.
  • Support administrative commands such as BGREWRITEAOF and ABORT and other commands.
  • Support commands that need access to store.
  • Support multi-key commands that potentially need access to multiple shards.

Reference: Worker Implementation for multi-threading support: https://github.com/DiceDB/dice/pull/539 Store refactoring for multi-threading support: https://github.com/soumya-codes/dicestorepoc

Note: We need to ensure that the corresponding unit/integrations tests are also updated accordingly.

soumya-codes avatar Sep 13 '24 10:09 soumya-codes

Please can this be assigned to me, I will sync with anyone that wants to collaborate on this

tdadadavid avatar Sep 13 '24 10:09 tdadadavid

Thanks @soumya-codes for the detailed breakdown, please let me know which part to pick. My logging work will be completed over the weekend, I'm good to pick a subtask from the list.

AshwinKul28 avatar Sep 13 '24 10:09 AshwinKul28

@pratikpandey21 is evaluating this already. Once he is done with the evaluation we will need many hands in this task. @tdadadavid I am sure there would me more than enough task on this to pickup.

@AshwinKul28 lets discuss on what part of the tasks you would like to pick-up.

soumya-codes avatar Sep 13 '24 14:09 soumya-codes

Hello @soumya-codes I can start with point 1 (Work on breaking down eval commands), We can talk about this, and I would appreciate your thoughts on how to approach it. If we need to wait for Pratik, I'm fine with that too, and then I can start on it.

AshwinKul28 avatar Sep 13 '24 17:09 AshwinKul28

@AshwinKul28 we can have a small chat so that we are aligned.

pratikpandey21 avatar Sep 13 '24 17:09 pratikpandey21

@soumya-codes Can I work on this, in collaboration with someone?

gauravrp avatar Sep 20 '24 08:09 gauravrp

Hey @gauravrp thanks for the interest. But this was completed, missed adding it under the PR and close it. You can check out the PR and keep looking for many upcoming multithreading issues.

AshwinKul28 avatar Sep 23 '24 21:09 AshwinKul28