Netflix
How to leverage DB join when retrieving data
Hi All,
So far, all the examples I have seen assume that there are no join being made by the database and most examples use services and expect the application to join.
If the router need to retrieve Foo.a and Foo.Bar.a given a Foo ID, how can this be done with only one round trip to the database ?
The router is not optimized for accessing a single database. What you need is a query optimizer which walks the tree of paths and builds a single SQL query (tree fold/visitor pattern-style). Router is specifically built for a REST or SOA backend (typical of a highly distributed and scaled architecture). This is Netflix's use case, which is why we have focused on it first. It is a good deal simpler to create a router than a query optimizer so it's the sweet spot if you have an SOA backend.
I understand it's not the sweet spot of Falcor right now given it's not a use case of Netflix with how you expose your data with highly distributed services where there is no DB joins at all.
That being said, the vast majority of apps and systems do not operate at Netflix scale and do end up using a form of datastore that in in most cases, allow some form of joins.
The query optimizer as you call it is not necessarily complex to do but once I have a partial graph of data, how do i inject back into the router ?
Right now my best bet seems to have some kind of cache for the duration of the http request where I can store nodes of the partial graph by path so that when the router recurse to retrieve the other paths, I can lookup into the cache to see if it's there before actually trying to fetch the data from the datastore.
I was wondering if maybe the router could provide the foundation for that cache and lookup approach.
The router seems to already have a cache but I dont know if it's something that the user is supposed to manipulate somehow.
My best guess is that the best way to approach this would be to accept a series and paths, mix them into a single map, and use a visitor pattern to build a SQL query. Not sure why you would need to cache in the circumstances, or anything provided by the Router. Would you be targeting a single SQL database?
Ya this is pretty much the patch I took.
However, if I am able from a top node to further fetch down the tree, how do I provide this information in a way that the router not gonna try to recurse the tree to resolve the childs ? Or is it that if the root node jsong have those childs already in its response, then the router will merge it with its cache and will fetch it from there directly ?
Don't use a router. If you're targeting a single SQL database, and you absolutely need a single query to it, just implement the data source interface. Then you can intercept the get method, collect up all the paths into a single tree, and write a visitor. The express middleware doesn't work with the Router, it works with DataSource Interface which the Router implements.
From Router docs
When to use a Router
The Router is appropriate as an abstraction over a service layer or REST API. Using a Router over these types of APIs provides just enough flexibility to avoid client round-trips without introducing heavy-weight abstractions. Service-oriented architectures are common in systems that are designed for scalability. These systems typically store data in different data sources and expose them through a variety of different services. For example Netflix uses a Router in front of its Microservice architecture.
It is rarely ideal to use a Router to directly access a single SQL Database. Application’s that use a single SQL store often attempt to build a single SQL Query for every server request. Router’s work by splitting up requests up for different sections of the JSON Graph document into separate handlers and sending individual requests to services to retrieve the requested data. As a consequence Router’s rarely have sufficient context to produce a single optimized SQL query. We are currently exploring different options for supporting this type of data access pattern in Falcor in future.
Thanks for the information @jhusain , I do see how the Router is shaped after aggregating services like its name implies :) Hopefully we will see a project emerges from the community to fill the gap and provide a solution for people who are just talking to a single datasource.