[FEATURE] Offer OpenSearch Dashboards APIs as Public APIs

[jkdll]

Is your feature request related to a problem? Please describe. In order to maintain a proper CI/CD pipeline, as a developer I would like to be able to use APIs to import and export OpenSearch dashboards objects such as Dashboards, Visualizations, Index Patterns and Tenants amongst others.

This would allow an engineer to write scripts which perform the necessary logic to move a dashboard from a development environment to a production environment seamlessly, and is particularly useful in use-caes where OpenSearch is being used to do analytics which go beyond logs.

This would also facilitate moving towards OpenSearch, from other Open Search-compatible NoSQL distros.

Describe the solution you'd like

Treating the APIs as public APIs, and having them documented and supported as such.

Describe alternatives you've considered

We are developing a proof of concept using "private" APIs using OpenSearch, there are also a number of third party terraform modules which attempt to do this. However, the fact that the APIs are not publicly supported is a blocker for most development in this area.

Additional context

Within my analysis, I found an area which was not obvious to me: https://github.com/opensearch-project/security-dashboards-plugin/issues/1002

[ashwin-pc]

@jkdll All OpenSearch Dashboard objects (dashboards, Visualizations and Tenants) are stored as documents on the OpenSearch cluster using the .kibana index.

e.g.

{
        "_index" : ".kibana_1",
        "_id" : "visualization:37cc8650-b882-11e8-a6d9-e546fe2bba5f",
        "_score" : 1.0,
        "_source" : {
          "visualization" : {
            "title" : "[eCommerce] Sales by Category",
            "visState" : """{"title":"[eCommerce] Sales by Category","type":"area","params":{"type":"area","grid":{"categoryLines":false,"style":{"color":"#eee"}},"categoryAxes":[{"id":"CategoryAxis-1","type":"category","position":"bottom","show":true,"style":{},"scale":{"type":"linear"},"labels":{"show":true,"truncate":100},"title":{}}],"valueAxes":[{"id":"ValueAxis-1","name":"LeftAxis-1","type":"value","position":"left","show":true,"style":{},"scale":{"type":"linear","mode":"normal"},"labels":{"show":true,"rotate":0,"filter":false,"truncate":100},"title":{"text":"Sum of total_quantity"}}],"seriesParams":[{"show":"true","type":"area","mode":"stacked","data":{"label":"Sum of total_quantity","id":"1"},"drawLinesBetweenPoints":true,"showCircles":true,"interpolate":"linear","valueAxis":"ValueAxis-1"}],"addTooltip":true,"addLegend":true,"legendPosition":"top","times":[],"addTimeMarker":false},"aggs":[{"id":"1","enabled":true,"type":"sum","schema":"metric","params":{"field":"total_quantity"}},{"id":"2","enabled":true,"type":"date_histogram","schema":"segment","params":{"field":"order_date","interval":"auto","drop_partials":false,"min_doc_count":1,"extended_bounds":{}}},{"id":"3","enabled":true,"type":"terms","schema":"group","params":{"field":"category.keyword","size":5,"order":"desc","orderBy":"1","otherBucket":false,"otherBucketLabel":"Other","missingBucket":false,"missingBucketLabel":"Missing"}}]}""",
            "uiStateJSON" : "{}",
            "description" : "",
            "version" : 1,
            "kibanaSavedObjectMeta" : {
              "searchSourceJSON" : """{"query":{"query":"","language":"kuery"},"filter":[],"indexRefName":"kibanaSavedObjectMeta.searchSourceJSON.index"}"""
            }
          },
          "type" : "visualization",
          "references" : [
            {
              "name" : "kibanaSavedObjectMeta.searchSourceJSON.index",
              "type" : "index-pattern",
              "id" : "ff959d40-b880-11e8-a6d9-e546fe2bba5f"
            }
          ],
          "migrationVersion" : {
            "visualization" : "7.10.0"
          },
          "updated_at" : "2022-05-20T05:01:19.706Z"
        }
      }

You can already access this using cURL, any of the api clients or even the devtools on Dashboards. Is there a specific use case that i'm missing that you think isn't covered by this?

[ahopp]

@jkdll any thoughts on what @ashwin-pc shared? Specifically, do you have any current use cases aren't covered by this?

[kilitr]

Hi!

Is it also possible to create completely new Tenants, Visualizations and Dashboards through writing documents?

Also i'm currently exploring my environment through those kibana indexes, however i don't recognize my tenants? How would i identify those on a document level?

[ashwin-pc]

@kilitr Tenants are a concept of the security plugin. I'll let someone from @opensearch-project/security answer that. But from what i know, each tenant is just a namespaced .kibana index. e.g. .kibana_<tenantID>.

As for creating Visualization and Dashboards, you can add a raw document to the correcponding .kibana[_tenant] index or use the saved object api.

e.g. to create an index pattern

curl 'http://localhost:5601/api/saved_objects/index-pattern/{index-pattern-id}' \
  -H 'Content-Type: application/json' \
  -H 'OSD-XSRF: true' \
  --data-raw '{"attributes":{"title":"osd-*"}}' \
  --compressed

afaik, if you pass the auth headers correctly, the saved object api should create the visualization and other documents that rely on the saved object service correctly.