chiselstrike icon indicating copy to clipboard operation
chiselstrike copied to clipboard

Sharding/Partitioning: How to?

Open 7flash opened this issue 2 years ago • 6 comments

In my application each user interacting with his own encrypted data synced across his devices, and therefore I need to have a new sqlite instance spin up for every user.. assume there are only a couple of users.. what's the easiest way to do it..

Considered alternatives:

  1. add user_id field to each model and filter by authenticated request..

  2. run multiple chisel processes on different ports

  3. single chisel process dynamically choosing underlying database file by endpoint.. there is usually /dev endpoint in the URL..

7flash avatar Sep 02 '22 09:09 7flash

If you absolutely require separate databases, then 2. is your only option right now. But if you trust ChiselStrike to keep the users out of each other's data, consider using the match_login policy.

dekimir avatar Sep 02 '22 22:09 dekimir

Another option.. partitioned tables.. can I create multiple CRUD endpoints using the same table model but unique rows?

7flash avatar Sep 04 '22 23:09 7flash

match_login seems a good option.. but also I need to implement "sharing" feature.. how can I modify it's transformer to NOT ignore those rows which has been made public by it's author?

7flash avatar Sep 05 '22 00:09 7flash

match_login seems a good option.. but also I need to implement "sharing" feature.. how can I modify it's transformer to NOT ignore those rows which has been made public by it's author?

We are evolving the policy language in that direction, stay tuned. :)

Another option.. partitioned tables.. can I create multiple CRUD endpoints using the same table model but unique rows?

Yes, but you'll also need custom code to implement the sharing usecase.

dekimir avatar Sep 06 '22 03:09 dekimir

@7flash The policy overhaul being done by @MarinPostma is happening in PRs like this: https://github.com/chiselstrike/chiselstrike/pull/1746

The main idea is that it will be native typescript instead of YAML, and then you will be able to express your custom logic. One use case we have in mind is to do both access control and geo-location (if users are in Europe, store data in a database that stays in Europe). I guess your use case is a variation of the latter, so it would work well.

Feedback on that work is appreciated!

glommer avatar Sep 06 '22 14:09 glommer

hello @7flash, the work on typescript policy is still very WIP, but happy to discuss your usecase!

MarinPostma avatar Sep 06 '22 17:09 MarinPostma