Unify list aggregation fields

[b5]

We have a number of places across where input parameters affect a listing result. While working on API refactoring (#1731) we've arrived at the following terms & fields for aggregation control:

name	data type	description
offset	int	how many elements to skip from the zeroith
limit	int	max number of resulting items
filter	?	reduce the resulting set according to one or more named filter functions
orderby	?	order items according to a set of named ordering functions

We've all agreed that selector is a separate concept that mapping into a data structure via a string path. While a selector may be used on an aggregation, it is not the same as a filter. Selectors will generally be used with individual dataset references.

It'd be great if we could write one package that defines aggregation fields, and an interface that each aggregator function can define filter & orderly functions for.

Steps to close:

This issue can be closed when:

• [ ] we've defined how filter & orderBy will work in practice
• [ ] all aggregation methods are enumerated here
• [ ] all aggregation methods accept the above control fields

[b5]

I'm seeing a nice chance for harmony coming from recent code spikes

@dustmop has a great PR #1746 that examines out how cursors could work in practice:

package lib

// MakeCursor returns a cursor that is able to retrieve the next page of results
func (s *scope) MakeCursor(nextPage interface{}) Cursor {
 	return cursor{s.inst, s.method, nextPage}
}

In a recent spec test I sketched out a list package that defines a list.Params struct:

package list

type Params struct {
 	Filter  []string
 	OrderBy []string
 	Limit   int
 	Offset  int
 }

if Params got a new method: NextPageParams() list.Params, we could define a new interface that list.Params would implement, and modify MakeCursor to accept it instead:

// Pager provides parameters for iterating an aggregation
type Pager interface {
  // NextPageParams returns a new list.Params that advances one page
  NextPageParams() list.Params
}

// elsewhere ...
func (s *scope) MakeCursor(pager list.Pager) Cursor {
 	return cursor{s.inst, s.method, pager}
}

Using an interface would let us do embedded-struct pagination:

package lib

type ListDatasetParams struct {
  list.Params  // embedding gives ListDatasetParams a NextPageParams method

  // these are all currently needed by listing datasets, here mainly to prove that listing
  // has case-by-case param fields:

  // ShowNumVersions only applies to listing datasets
  ShowNumVersions bool
  // Raw indicates whether to return a raw string representation of a dataset list
  Raw bool
  // ...
}

We lose the explicit declaration of each field on the struct with embedding, but I think the wins here are worth the cost. The main thing I'd like to point out: Cursors and Pagination parameters are related concepts we should take steps to coordinate

[Arqu]

Another thing that came up while inspecting our routes is that a generic filter/orderby param is not sufficient to know what the value is referring to. A good example would be /list which will no longer handle the sugar route /list/{peer} and will be absorbed as /list?filter=some_peer.

This works for endpoints with a single filter option but not with those that support multiple options. /list can filter by ProfileID, Peername, Term, Public and possibly other flags. How do we decide which get a top level parameter like ?public=true vs a generic filter. Also opens the question of accepting multiple filter/order key value pairs for multi-key filtering.

OpenAPI, as far as I know, supports several schemas to encode multiple values but the common one I've seen is along the lines of ?filter[peername]=some_peer&filter[public]=true which would be received as a map[string]string on the API side and processed further (we could magically just put this into the r.FormValue similar to refstr right now).

This would look like: map[string]string { "peername": "some_peer", "public": "true", }

[Arqu]

Copy of some discord input from b5:

and more specifically, how do we specify filter key-value pairs? multiple filters? graphql has a very verbose syntax for filtering, but it's super robust:

query {
  getAuthor(id: "0x1") {
    name
    posts(filter: {
      title: {
        allofterms: "GraphQL"
      }
    }) {
      title
      text
      datePublished
    }
  }
}

I'd psudocode that in go as:

type Filters map[string]Predicate

type Predicate struct {
  AllOfTerms string
}

which would express in JSON as {"filter": {"username": {"allOfTerms": "arqu" }}} we could add sugar to Predicate struct deserialization that says "if I'm a string, interpret that as { "allOfTerms" : [string] }" which gives us {"filter": {"username": "arqu" }}