juniper icon indicating copy to clipboard operation
juniper copied to clipboard

Document N+1 mitigation techniques in the book

Open MikailBag opened this issue 4 years ago • 6 comments

Is your feature request related to a problem? Please describe. Now, resolver signature looks like: fn field(&self, /*args*/). This can easily lead to N + 1 DB query problem. (Or even N**K if query has K levels of depth). To avoid this problem, special libraries should be used, that somehow accumulate queries and merge them into one.

Describe the solution you'd like (Idea itself is very simple. It is possible that is was already discussed and rejected, but I didn't find such an issue). I propose to allow batched resolvers. Signature now looks smth like: fn field(self: &[Self], ctx:&Ctx, args: ...) Now, from N**K requests we go to K (each level will be processed with one resolver execution and one db request). Describe alternatives you've considered Alternative: do nothing. However, I believe batched resolvers provide good enough performance with high flexibility.

MikailBag avatar Oct 28 '19 21:10 MikailBag

@MikailBag for N + 1 query problem the usual solution is either dataloader or lookahead.

As for proposed solution I'm unsure how well it will fit. Usually, resolvers may contain allowed-resoling-depth checks, authorization-rights check, authentication checks, etc, which all are query-scoped. Resolving multiple queries at the same time, but now with the same context, seems to be less flexible.

tyranron avatar Oct 29 '19 07:10 tyranron

If you're interested in the lookahead approach you could consider juniper-eager-loading which handles all the boilerplate. We're using it in production at @tonsser and its been working well so far. Full disclosure: I'm the maintainer of the library.

I'm sure once async support has landed that we're gonna experiment with the dataloader approach as well.

davidpdrsn avatar Oct 29 '19 08:10 davidpdrsn

As mentioned by others, the possible approaches here are

  • datalaoder
  • using lookahead
  • uniper-eager-loading

We should really add to the book, since it's a common stumbling block.

theduke avatar Oct 29 '19 10:10 theduke

Thanks for the responses! I will study your proposals (so far juniper-eager-loading looks the most promising).

Regarding book, I would like to see quite complex example, e.g.:

  • Response data are aggregated from DB and in-memory source (e.g. calculated).
  • Application doesn't directly use diesel, it uses some trait, MyAppDbProvider.
  • And without too much glue code :)

MikailBag avatar Oct 29 '19 11:10 MikailBag

@MikailBag I maintain a complete example app. You can find it here https://github.com/davidpdrsn/graphql-app-example. It does use Diesel directly however.

davidpdrsn avatar Oct 29 '19 12:10 davidpdrsn

@MikailBag I'd be happy to make any requested changes to my demo repo and provide it as an example.

https://github.com/jayy-lmao/rust-graphql-docker

It currently uses dataloaders and has an emphasis on code structure. I welcome any feedback/critique.

Edit: It uses actix-web, juniper, postgres, dataloader-rs

jayy-lmao avatar Jan 29 '20 13:01 jayy-lmao