Projected Views

[peternied]

Description

A new kind of resource that abstracts indices from searches. A view define a projection of targeted indices. I'm trying to start out with as barebones of an implementation to get design feedback and find a good matching vertical scenario that could be added to dashboards.

Related Issues

• Resolves https://github.com/opensearch-project/security/issues/3888
• RFC https://github.com/opensearch-project/OpenSearch/issues/6181
• Based off https://github.com/peternied/OpenSearch-1/pull/139

High Level Design

Views are stored in cluster metadata, standard CRUD operations. A view can be used for _search which validates and transforms the request onto the 'physical' structure of the data like indices. See more in the views-design.md [link] included in this PR.

sequenceDiagram
    participant Client
    participant HTTP_Request as ActionHandler
    participant Cluster_Metadata as Cluster Metadata Store
    participant Data_Store as Indices

    Client->>HTTP_Request: View List/Get/Update/Create/Delete<BR>/views or /views/{view_id}
    HTTP_Request->>Cluster_Metadata: Query Views
    alt Update/Create/Delete
        Cluster_Metadata->>Cluster_Metadata: Refresh Cluster
    end
    Cluster_Metadata-->>HTTP_Request: Return
    HTTP_Request-->>Client: Return

    Client->>HTTP_Request: Search View<br>/views/{view_id}/search
    HTTP_Request->>Cluster_Metadata: Query Views
    Cluster_Metadata-->>HTTP_Request: Return
    HTTP_Request->>HTTP_Request: Rewrite Search Request
    HTTP_Request->>HTTP_Request: Validate Search Request
    HTTP_Request->>Data_Store: Search indices
    Data_Store-->>HTTP_Request: Return
    HTTP_Request-->>Client: Return

Check List

• [x] New functionality includes testing.
  • [x] All tests pass
• [X] New functionality has been documented.
  • [X] New functionality has javadoc added
• [X] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• [X] Commits are signed per the DCO using --signoff
• [X] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• [X] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

Compatibility status:

Checks if related components are compatible with change 2ac6e68

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

[github-actions[bot]]

:x: Gradle check result for 61e8c988751c413791805e632da9ef6ade71573f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[peternied]

@msfroh I'm going to keep this in draft until that checklist in the description has been completed, but this PR would be in a good place for feedback around the metadata changes / CreatViewAction / ViewService. Feedback would be greatly appreciated - thanks!

[github-actions[bot]]

:x: Gradle check result for 2f615cf14da2ae9422d56943942b3823b4d25122: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[peternied]

@cwperks @DarshitChanpura could you take another look? I've snapped to the patterns associated with the OpenSearch repo and created integration tests. While I still need to add some additional transport actions and associated tests (Get/Deleted/Update/Enumerate) they will follow the same conventions established with CreateViewAction and the ViewService.

[github-actions[bot]]

:x: Gradle check result for d45fcb933047c59dd894777c95eca50c7c83bb8b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 2832b24ca12fd192f7e41ee63ed5dbe751d07387: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 12296ce066e05004a9510bb437c401613a0574cf: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for a4a058a55619d54b664573281545813b1e2182bd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[peternied]

Docs PR:

• https://github.com/opensearch-project/documentation-website/issues/6363

[peternied]

@reta @msfroh @mch2 Mind taking a look?

[github-actions[bot]]

:x: Gradle check result for dd4c70259d6e1a89ef49143f94af47fd88a1e073: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 15a3a5061dd583d1037af2c7ca69fb4fd82724ee: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 242ab859daeae809982f730875e36df036c22fb5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[reta]

@reta @msfroh @mch2 Mind taking a look?

@peternied I should be able to look tomorrow, if that works for you, thanks!

[github-actions[bot]]

:x: Gradle check result for ea78012a81f51e8c21bb54a321f74e38b03a2297: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 07bcfe0be616ddf1ba93e7c12bfe2ee78584ad44: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for e7f36b0bb5e4fa57f583b2e5f240cc3c91df552c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 65fa8e7e869577f6c4798fdbdcb529977d81cf39: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 43272377801d88be3338c8b966fc88e5cb504139: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 5beefbbe55cb1f0132810a91da9f910ddafe5159: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 16b2b523da661e92fb4d26c7c4bbcd4e7ce2b479: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 16b2b523da661e92fb4d26c7c4bbcd4e7ce2b479: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for edd890264e5eb0e816945ffe01a9b7a8f784ab81: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for ba385e4ac5c6ebc170acae994cdbb29cb88b378d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 0663818c98d90b00cc6edc9aae957d9275af8b56: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 7f28330201c6bc1cbd925842ccbf975fc370b32c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 8cb9f55490d72ba4474f0bd97ead04298ef88439: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 6e9dffb15c5d10a8e71bbbf7400317538ef08f6d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for cba189e11c38dd8c807857ca12838944e681d77b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?