Filtering data for blocks is stateful API, introduce stateless data filters for API usage.

[mattsmithies]

Problem description

To provide context to the present DOVU approach, we are designing our policies to effectively force guardian to be more RESTful when to enable us to eliminates any N+1 problem when processing data through blocks

As an example, for our ELV credits we initially didn't have any pagination or filtering of data. It became incrementally more resource intensive to process/manage more entities. Including pagination by default, doesn't necessarily solve this problem, because we would need to create logic to entry through particular pages.

So, I've been looking at the filter block so I am able to call a data block directly to receive a particular element I am looking for, currently our schemas include a uuid at the source of the schema, we do this as we know we would have an externally controlled identifier that we are able to reference within guardian and scan for data updates, for a bit more context the current strategy for us consuming Guardian is as follows:

• Send data to a block (as a supplier)
• Scan for the data (as a registry) "approving a supplier" -- using the external uuid as a pointer.
• Then process the approval / or additional block submission with the related ref.

I've recently been working with the filter block, and what I found. Is would hypothetically allow me to construct more of a consumable (RESTful) interface to scan/pluck a single resource -- but the current filter flow is stateful (I also noticed that if you try to use tags instead of the block id the "filterValue" is set but ignored.

The concern here is that if there are 2 or more instances of attempting to filter the "filterValue" can collide and therefore you would have to double check that the value returned was what was expected.

(minor issue) Furthermore, it seems that there are 2 methods to update the filter, either:

• Through tag
• Through block

The block/tag API call updates the filterValue and subsequent calls to the documentSource block will provide the filtered result, in addition, the actual filtered value doesn't seem to be updated when using the API. See below:

  public function filterByTag(string $policyId, string $tag, string $uuid): object
  {
      $block = $this->fromTag($policyId, $tag);

      return (object) $this->httpClient->post("policies/{$policyId}/blocks/{$block->id}", [
          'filterValue' => $uuid,
      ], true);
  }

And on logging

So while the filterValue was set to "developer2" it didn't appear in the state -- which I presume means that it is connected else where.

Requirements

I don't necessarily think there is a hard requirement to remove the stateful nature of guardian filtering, as we cannot predict, what are the downstream API consumers are using this functionality or affects, they will be without some kind of deprecation notice.

So, the recommendation would be:

• Add ability to filter using a GET request for a filter, so data can be fetched and filtered in one action
• (As an alternative - preferred) It would be preferable to enable filtering at the block level when retrieving data so a API consumer does not need to add explicit filter blocks in block can use the Guardian API to be more RESTful by default.
• Post a six month deprecation notice for stateful usage of the filter (revert if hard requirement for others)

An example, code enhancement could be implemented like this (tags are easier to reason about):

From old version:

  public function filterByTag(string $policyId, string $tag, string $uuid): object
  {
      return (object) $this->httpClient->post("policies/{$policyId}/tag/{$tag}/blocks", [
          'filterValue' => $uuid
      ], true);
  }

to:

public function filterByTag(string $policyId, string $tag, string $uuid): object
{
    return (object) $this->httpClient->get("policies/{$policyId}/tag/{$tag}/blocks?filterValue={$uuid}");
}

Or provide/document clearly a mechanism to filter on an interface document block itself, which would be preferred.

Definition of done

Provide a stateless alternative for filtering block data in realtime where a API consumer can trust that returned data is the expected value.

Acceptance criteria

• Implementation of feature
• New test file of feature, a scenerio where:
  • Policy is imported
  • Policy enters dry run
  • 2 users are created
  • Each user submits a document (project)
  • The approval user (registry) can filter on a parameter to prove the filter working
  • Dry run is switched to draft
• Documentation that highlights usage of stateless filtering

[mattsmithies]

@anvabr @prernaadev01 Here is the issue for filtering, if possible, I would love to see filtering capibility on the InterfaceDocumentsSourceBlock itself.

[prernaadev01]

@mattsmithies Thank you so much for raising the ticket. We will discuss it internally and keep you posted on it.

[anvabr]

From the discussion with @mattsmithies, the following is a potential way of addressing this issue without having to deprecate/change current API

"If adding filters by default is too much of an issue/cost I believe being able to fetch a single item “data” by its id/uuid would be highly beneficial and simpler, and may get around downstream application effects of change through an addition of a new API route (if this already exists, please let me know). In this case the signature of the API route for a single resource could be something like this:"

GET `policies/{$policyId}/tag/{$tag}/blocks/data/{$data_uuid}`  &
GET `policies/{$policyId}/blocks/{$uuid}/data/{$data_uuid}`