Add an endpoint for querying public governance tables

[eddyashton]

To avoid manually defining wrapping endpoints for everything we want to expose, we should have a generic GET endpoint for anything in a public governance table. Should be fetchable by key, and the entire table. Requires some static knowledge of tablename-to-tabletype for these tables.

I think ideally this would look like:

GET /gov/kv?public:ccf.gov.nodes

But that has awkward encoding implications (colons in paths). A simpler alternative might be:

GET /gov/kv/nodes

This assumes every table we care about is prefixed public:ccf.gov., so we can manually add that.

Also not tied to /gov/kv, maybe /gov/everything.

[msftsettiy]

There is an ask from the field to view the constitution used in a network. This endpoint will be helpful to satisfy such asks and will improve the network transparency.

[eddyashton]

Took a quick look at this.

• Colons in URL paths are apparently safe, so I think we should support the full table name (and maybe allow easier suffixes as well for now)
• Most keys, but not all, can be parsed and presented as ASCII. Perhaps misleading for the unit value keys though (actual key is uint64_t 0, parsed as ASCII that's an empty string, but key is actually unequal with many other empty strings)
• So I think we offer query parameters to dictate the presentation format of keys and values (ascii, json, hex, base64). Note that this converts them all to strings (or JSON values), and those are then embedded into a JSON object
• Querying for specific keys is tricky for the same encoding reasons. Maybe we can use the same parameter to decode to raw bytes? Ideally we'd present raw key bytes in the body, but GETs are bodyless

Putting that altogether, I think this API looks like:

GET /gov/kv/public:ccf.gov.nodes?key_format=ascii&value_format=json

// GET /gov/kv/nodes would return the same value
// because these are the default query parameters, and we'll try to expand the table name

{
  "map": "public:ccf.gov.nodes",
  "contents": {
    "01abcd23ef3456ff": {
      "cert": "...",
      ...
    },
   "12ab12ab12ab": {
      "cert": "...",
      ...
   }
  }
}

GET /gov/kv/public:ccf.gov.nodes?key_format=ascii&value_format=json&key=01abcd23ef3456ff

{
  "map": "public:ccf.gov.nodes",
  "contents": {
    "01abcd23ef3456ff": {
      "cert": "...",
      ...
    }
  }
}

[jumaffre]

I think automatically converting a simpler slash-separated path to the actual name of the KV table will greatly improve adoption of this new generic endpoint. For example, GET /gov/kv/nodes is more concise and RESTful than GET /gov/kv/public:ccf.gov.nodes (as you said, colon in the URL is a little weird) and it will force us to follow this logic for names of new tables, which I think is a big win.

[eddyashton]

I think automatically converting a simpler slash-separated path to the actual name of the KV table will greatly improve adoption of this new generic endpoint. For example, GET /gov/kv/nodes is more concise and RESTful than GET /gov/kv/public:ccf.gov.nodes (as you said, colon in the URL is a little weird) and it will force us to follow this logic for names of new tables, which I think is a big win.

I'm wary of doing something slightly RESTy that impacts discoverability/legibility. GET /gov/kv/nodes looks nice, so nice it could have been hand-written, but then it's weird that it has such an awkward schema in OpenAPI, and it's non-obvious that for more information you should be looking for docs about a table named public:ccf.gov.nodes. For many governance tables, the name is already less nice - GET /gov/kv/members.info is a little odd, we'd prefer GET /gov/kv/members/info, but that's not currently supported and makes the mapping to a table name even more obtuse. I think there's an argument that this should be able to serve public:ccf.internal.* tables as well, and maybe-maybe just all public:* tables?

I think a direct, explicit inclusion of the table name makes it clear what this endpoint is doing (serving raw table contents), and that may be more helpful than making it look like auto-generated wrappers for all KV tables.

Maybe table name in the URI is nasty, so we move it to a query param:

GET /gov/kv?table=public:ccf.gov.members.info?

I think this needs multiple query params and is cumbersome to write by hand anyway, so this isn't a big change.

[heidihoward]

Found this ticket (+https://github.com/microsoft/CCF/issues/4170) whilst looking for an endpoint to get the current constitution. This would also be helpful to me for local testing of constitution changes.

[eddyashton]

WIP branch for this is here: https://github.com/eddyashton/CCF/tree/raw_kv_endpoint

I'll take another look at this today.