Best practice for exposing a RESTful collection through Falcor

[jmerrifield]

I work with a large number of APIs exposing collections of items. Typically these accept an offset/limit pair, and return a complete list of items (rather than a list of links to each item), as well as the total number of items available. For example:

/todos?offset=10&limit=10 might return:

{
  "total_count": 13,
  "items": [
    { "id": 100, "name": "get milk", "completed": false },
    { "id": 101, "name": "get bread", "completed": true },
    { "id": 102, "name": "get cheese", "completed": false }
  ]
}

Additionally /todos/101 would return the same data for a single item: { "id": 101, "name": "get bread", "completed": true }.

Exposing this API through Falcor routes should, if I'm following the docs correctly, look something like:

{
  route: 'allTodos[{ranges:indices}]',
  get({ indices }) {
    const { first, last } = materializeRange(indices);
    fetch(`/todos?offset=${first}&limit=${last - first}`);

    // for each number in `indices`, return a $ref to the corresponding TODO item, e.g:
    return [
      { path: ['allTodos', 10], value: $ref(['todosById', 100]) },
      { path: ['allTodos', 11], value: $ref(['todosById', 101]) },
      { path: ['allTodos', 12], value: $ref(['todosById', 102]) },
    ]
  }
},
{
  route: 'todosById[{integers:ids}][{keys:attrs}]',
  get({ ids, attrs }) {
    // for each id: fetch(`/todos/${id}`)

    return [
      { path: ['todosById', 100, 'name'], value: $atom('get milk') },
      { path: ['todosById', 100, 'completed'], value: $atom(false) },
      { path: ['todosById', 101, 'name'], value: $atom('get bread') },
      // etc...
    ]
  }
},
{
  route: 'allTodos.length',
  get() {
    // Make a request with arbitrary params to /todos to get the total count
    const response = fetch('/todos?offset=0&limit=1');

    return [
      { path: ['allTodos', 'length'], value: $atom(response.total_count) }
    ]
  }
}

The problem is that a typical usage of these routes:

falcor.get(
  ['allTodos', { from: 10, to: 19 }, ['name', 'completed']],
  ['allTodos', 'length']
).then(console.log);

Would trigger (in this example) 5 API calls when just 1 would suffice - /todos&offset=10&limit=10 has all the necessary data to fulfill this entire query.

I'm currently using 2 different approaches to deal with this:

To share data between the 'list' and 'detail' routes (i.e. allTodos and todosById) I'm using a mutable cache object on the router. In allTodos I store 'extra' data for use later:

// allTodos
return requestedIndices.map(i => {
  const item = response.items[i - offset];

  // Remember values for later  
  _.set(this.cache, ['todosById', item.id, 'name'], item.name);
  _.set(this.cache, ['todosById', item.id, 'completed'], item.completed);

  // Just return a $ref
  return { path: ['allTodos', i], value: $ref(['todosById', item.id]) };
});

// todosById
return ids.map(id => {
  // for each requested attribute ('name', 'completed'), see if we have it in `this.cache`
  // if all the requested attributes are in cache, don't make an HTTP request, just return them:
  return [
    { path: ['todosById', id, 'name'], value: _.get(this.cache, ['todosById', id, 'name']) },
    { path: ['todosById', id, 'completed'], value: _.get(this.cache, ['todosById', id, 'completed']) },
  ];
});

To fulfill the length value without making a separate request, I think I can just return that as an extra (un-requested) path value from the allTodos[{ranges:indices}] route:

// allTodos
return [
  { path: ['allTodos', 10], value: $ref(['todosById', 100]) },
  { path: ['allTodos', 11], value: $ref(['todosById', 101]) },
  { path: ['allTodos', 12], value: $ref(['todosById', 102]) },

  { path: ['allTodos', 'length'], value: $atom(response.total_count) }
]

And as long as allTodos[{ranges:indices}] appears before allTodos.length in the route list, then the allTodos.length route should not be invoked (but requesting allTodos.length in isolation would still invoke the route and work properly).

I'm looking for guidance or best practices on dealing with this scenario. Are my 2 workarounds reasonable or likely to cause problems later? Does anyone have a better way of handling this situation? Am I correct in my choice of Falcor routes to expose a collection of items?

[ludovicthomas]

On our side, to avoid adding some cache logic, we have added the object attributes to the allTodos route, so a single call is done to get refs and properties. We use shared methods between allTodos and todosById routes to ensure consistency between attributes returned.

Here is a sample in your case:

{
  route: 'allTodos[{ranges:indices}][{keys:attrs}]',
  get({ indices, attrs }) {
     ... return $ref and prop of todosById...
  }
}

For the length route, we have a separate route in our API, so there is no need for optimization on this one on our side.

Would be happy to have some feedback of other Falcor devs too.

[jmerrifield]

Thanks @ludovicthomas, that's really interesting! I think I prefer your approach - just so I'm clear, is your allTodos route returning the following?

// Assuming we asked for allTodos[0]['name']
[
  // $ref to todosById
  { path: ['allTodos', 0], value: $ref(['todosById', 100]) },

  // values _under the todosById path_
  { path: ['todosById', 100, 'name'], value: $atom('get milk') }
]

[ludovicthomas]

Sorry for the late reply.

Yes that's exactly that, in fact we have helpers that we use on both routes todosById and todos to return the properties of todosById. It's called todoToPathValues in the following example

Something like this for route todos:

// Get data from service call, then construct paths
const properties = pathSet.attrs;
let pathValues = _.map(pathSet.indices, (indice, index) => {
    const todo = data[index];
    let results = { path: ['todos', indice], value: $ref(['todosById', todo.id])};
    results.push.apply(results, todoToPathValues(todo, properties));
    return results;
});
return _.flatten(pathValues);

and something like this for the route todoById:

// Get data from service call, then construct paths
const properties = pathSet.attrs;
let pathValues = _.map(pathSet.todoIds, (id, index) => {
    return todoToPathValues(data[index], properties);
});
return _.flatten(pathValues);

With the todoToPathValues something like:

function todoToPathValues(todo, properties) {
    let results = [];
    properties.forEach((key) => {
        results.push({
            path: ['todoById', todo.id, key],
            value: _.get(todo, key, $atom(null))
        });
    });
    return results;
}