[UX] Point in time in OpenSearch Dashboards (WIP)

[Cindyqi8506]

Overview

OpenSearch Dashboards Point in time (PIT) search feature allow users (including but not limited to Developers, DevOps, and Data Analyst) to create and manage point in time search from OpenSearch Dashboards stack management, discover, visualize and dashboard. It helps user maintain a fixed data set which can be re-used by different queries in order to achieve consistent results.

Use Cases

1. Users create a PIT based on existing index pattern or indices, give the PIT a name and a expiration date.
2. Users explore the existing index pattern in Discover, and create a PIT to get fixed data set in time.
3. Users create visualization based on the existing index pattern in Visualize, and create a PIT to freeze the visualization result and share with others.
4. Users view and manage PITs in stack management, perform tasks such as extend expiration time, bulk delete, rename and create new PIT based on existing ones.

We are looking for more use cases, please comment if you know a use case not covered above. 😃

Ref: Point In Time Search opensearch-project/OpenSearch#1147

Flow structure

There are four flows in exploration for PIT search, Minimum viable product (MVP) will include Stack Management flow and Discover flow. Visualize flow and Dashboard flow are working in progress (WIP).

Stack Management Flow (MVP)

• From Stack Management > Point in Time > view and manage PITs
• From Stack Management > Index Pattern > create PIT based on existing index pattern
• From Stack Management > Saved Objects > relationship (parent/child), view PITs and its’ relationship with other objects 

Discover Flow (MVP)

• From Discover > Change Data Source > create PIT or change to existing PIT

Visualize Flow (WIP)

• From Visualize > Change Data Source > create PIT or change to existing PIT

Dashboard Flow (WIP)

• From Dashboard > Edit Panel > create PIT or change to existing PIT / Show expired PIT Panel

Design Mockups

1. Stack Management Flow (MVP)

• Access Point in Time in Stack Management

• Create new PIT in the flyout

• Perform tasks such as extend expiration time, bulk delete, rename and create new PIT based on existing ones.

• View PIT Details 

• Create new PIT based on existing index pattern

• View PIT in saved Objects

2. Discover Flow (MVP)

• Switch data source to existing PIT
• Create New PIT based on selected index pattern

3. Visualize Flow (WIP)

• Switch data source to existing PIT created based on the selected index pattern
• Create new PIT based on selected index pattern

4. Dashboard Flow (WIP)

• Create new PIT for the selected panel
• Click PIT icon to open the PIT flyout and edit
• Expired PIT icon turn red and graph become monochrome (we may not have the expired PIT data, consider empty state pattern instead)

Thank you for reading, comments and suggestions are welcome 🤗.

[ahopp]

Hey @Cindyqi8506 this is great! Couple of questions;

RE: "Create new PIT based on selected index pattern" within the "Visualize Flow", how do you imagine the "change PIT" capability to work when there is 2 dozen PITs available? I'm trying to imagine how this will scale overtime and I think we should annotate some of our thoughts if we're still WIP for this scope. Similarly we need to handle the "Create New PIT based on selected index pattern" CX with similar regards to scaling.

RE: "Expired PIT icon turn red and graph become monochrome (we may not have the expired PIT data, consider empty state pattern instead)", do we now if we will even have enough cached in the case of expired PIT data? Or is that even possible? Either way, I think we should have an empty state CX that provides users a direct path to replace the data source or adding a new visualizations. What do you think?

RE: Expired PIT actions, is there anyway we can make it more clear that the actions aren't available on expired PITs? It looks like we've take a "lighten" approach (which feel like a good start) but I find myself wondering if that serves any value to the users? Why see what you can't do? How do we feel about dropping those actions all together? Is this because the "lightened" pattern (i.e. lighten the action buttons on the screen when unavailable) is held across other part of the UI?

These are a few of my thoughts, but the CX looks great so take my feedback with a grain of salt!

[dblock]

Moving this to https://github.com/opensearch-project/OpenSearch-Dashboards.

[kavilla]

[Groom]:

@kavilla will follow-up about ownership about this.

[ahopp]

@kavilla RE: ownership, I think the best model of ownership is the PIT feature devs owning the PR and the Dashboard repo owners offering support where we can and to helping with any primitives that would be broadly usable by Dashboards in the future - especially given the deep Dashboards backlog atm.

[ashwin-pc]

Maybe im not reading this correctly, but how does this differ from saved searches or saved queries?

[kavilla]

@ashwin-pc

I believe on the data source side it would be essentially a view at the data at that specific time compared to timestamps of the data.

Like a snapshot of the data. If I remember the implementation details on the engine side completely partitioned from the data that OpenSearch Dashboards normally hits.

[kgcreative]

Let's seek some additional clarity on implementation here. I think after sitting some time with this, it feels like we're creating a front-end abstraction that creates permanent objects to track ephemeral resources. Perhaps a different mechanism (PIT id in the URL when creating a dashboard snapshot?) might be a better mechanism for this. I'd like to see some additional thought put into how we might abstract some of these implementation details from the user, where they don't have to track correlated objects in order to get value out of PIT. This seems particularly well suited for some abstract use cases such as pagination, etc.

[kgcreative]

Empty state:

[ashwin-pc]

@kavilla thanks for the clarity.

My biggest concern with this feature in Dashboards is how its integrated with another similar but purely dashboard feature "Saved Search". Judging by the design here, it looks like we want to give the user granular control to constrain data that a particular feature (e.g. visualization, discover) uses for a given index(s). For this feature its a point in time. for saved Searches it is the query, timerange and filters.

@Cindyqi8506 How does this feature work in relation to Saved Searches? Visualizations today can be created using either an index pattern or a Saved Search and you can also add a Saved Search to Dashboards (Both are features proposed in this UX). They also have a similar saved object relationship with index patterns as PIT objects. And ideologically also have a similar functionality, to constrain the index data:

• Saved Searches constrain based on search parameters
• PIT constrains based on timestamp

The UX here worries me a lil because the two features have very different user flows but are quite similar ideallogically and in where they are used. Having two completely different ways to manage and use these properties can become confusing to both user and developer's who want to integrate these features into other parts of OSD. i.e. Dashboards, Visualize, Plugins. e.g. In Discover, we have the "New, Save and Open" (Screenshot below) options on the top nav for managing Saved Searches but in this UX we are proposing that we have a completely different page to manage PIT objects.

This also extends to how PIT is used. Saved Searches sit beside Index patterns in the datasource picker in visualize but in the UX here saved searches are no longer included in the datasource picker. And on the Dashboard, we can directly add a Saved Search to the dashboard (similar to a visualization) but a PIT is created against a visualization (This actually seems redundant since we can already pick to create a visualization using a PIT object).

I think we should combine the Saved Search and PIT flows to be more cohesive so that a user who wishes to constrain their search either by timestamp or by search parameters (or ideally both) should have a similar flow and developers who wish to support these integrations in their plugins have a similar api. Otherwise while the feature might exist, its rollout will be spotty in different parts of the app. And who's to say that this is where we stop constraining. In future, we may also want to constrain based on datasource (now that we are introducing multiple datasources)

cc: @kgcreative

[kgcreative]

Hi @ashwin-pc -- Here's my thinking around how PIT correlates to other dashboard object constructs, and ways we might be able to continue to potentially abstract or enhance the feature.

For this feature its a point in time. for saved Searches it is the query, timerange and filters.

PIT is orthogonal to saved search, and might be more closely related to snapshots. From a back-end perspective, you can think of a PIT as an ephemeral index snapshot. Given a PIT ID, it preserves the state of an index at the time of creation. The challenge we were faced as we were thinking about how to integrate this in Dashboards, is that we don't have a way to direct a saved search to reference an ephemeral resource without creating some kind of abstraction to track it. (I'm actually wondering if PIT is the correct feature name, or if there is a better name).

Ultimately, we felt that PIT having a child relationship to an Index pattern made the most sense, particularly since this would allow us to fall-back to the source index pattern if the PITs in the saved object have expired and are no longer available in the cluster.

Some additional options we explored is instead use the "Share" mechanism to create a "Point in time snapshot" that has a 24h expiry, which might just create an ephemeral object that tracks the underlying index states. That might end up being a better abstraction long term, but we haven't fully reasoned about some of the other use cases.

Ultimately, it seems like the primary use for PIT might be things like Notebooks, and Discover, but less so saved searches or other more "permanent" objects, since PIT IDs (in the backend) are ephemeral and will not persist beyond a cluster restart.

[ashwin-pc]

@kgcreative That makes sense. From that perspective the difference between "Saved searches" and "PIT" is clearer (and yes, the name could be better). But i'm still not very happy with how we are integrating this feature into the different apps. I love how Multi Datasource solved this problem by being transparent to the apps that relied on the core services. To the visualize, dashboards and other apps, its practically non existent. Doing something like that here would help.

For example. if in the PIT management screen we could also added a toggle to enable a PIT. All visualizations, discover searches and Dashboards that used index patterns that link to the enabled PIT would receive data for that snapshot. This means that other apps e.g. VisBuilder would also work out of the box with no additional modifications. (And maybe other external plugins too!).

If we want to make toggling between PIT instances easier, we could expose an option in the top nav to toggle this, similar to how we toggle between tenents.

And finally , if we want more granular control in certain apps e.g notebooks, they can have custom integrations for PIT that lets them for example set specific PIT for each notebook section. But that does not limit other apps from benefitting without any modification.

[kgcreative]

@ashwin-pc In the fullness of time, we need full integration with Discover, Saved Search, Dashboards, Visualizations, and any other saved object that can interface with Index Patterns. I think we're basically just looking at what we can deliver incrementally to start delivering value, which in this case, being able to integrate with Discover solves the short term need of being able to interact with ephemeral data directly. I think there are a lot of opportunities to consider how we might use PIT searches transparently under the hood without potentially even the need for this extra abstraction, and it would be really good to start exploring them

[kgcreative]

(For example, we could leverage PITs for cached results and pagination transparently under the hood if the backend feature exists, and we could track that temporarily for some ammount of time, which could greatly increase performance for some things)

[ashwin-pc]

Talked to @kgcreative offline. I think what we need here are we need user stories. What are the user problems we are trying to solve. My personal opinion is that PIT is a very interesting feature. But we should be careful to not use Dashboards as a kitchen sink to demo that feature. We have to also think about how this feature will integrate with other dashboards projects like different datasources and PPL.

An example user story might be that as a user I would like to pause my data for a moment while using a dashboard to study the data further. This lets us be more precise with our integration in a cohesive and scalable way. E.g. in such a usecase the UI could simply be a button that says "Pause" on the top nav for example and you wouldnt even need to create any custom integration with dashboards, the data plugin would handle that interaction.

[bharath-techie]

@kgcreative @ashwin-pc @kavilla , We are gearing up for actual development of 2.5 milestones that we've kept for Point in time UX changes for which we did the POC.

Can we finalize on the scope for 2.5 ? There are lots of great points mentioned above, but some are quite different from POC - so it'd be great if we can get a direction on this.

[bharath-techie]

Hi. So far, for 2.5, this is the scope that we've planned.

Area	Item
Stack management	New plugin changes
	Table changes
	Create PIT Side drawer
	Rename PIT option
	Success / failures
	Showing PIT details screen - general tab
	API integration - List pits, delete pit and create PIT
	Delete PIT after expiry - data plugin
	Plugin experimental flag changes
Saved objects	PIT option in saved objects section
Discover	Support for PIT search
	Change index pattern component changes
	Point in time cache
	Top section - new, save, share, inspect flows
	Side drawer integration
Misc	Documentation
	Security

Let us know if there will be any changes to this - @kgcreative

[joshuarrrr]

@bharath-techie Do you mind helping us organize and label the PIT-related issues? I've labeled them all feature-PIT for now and created an Epic to track them all: https://github.com/opensearch-project/OpenSearch-Dashboards/issues/2909

It seems like the items in this table are more up to date than the task lists in each issue. Can you please

1. review the existing issues and update the task lists accordingly
2. add the v2.5.0 label to any issues that are targeted to be fully complete for that release

As for the feature branching process, please refer to https://github.com/opensearch-project/OpenSearch-Dashboards/issues/2887. For each incremental PR, it's helpful to have a corresponding issue that describes the specific functionality that PR will add.

[kgcreative]

cc: @KrooshalUX for FYI

[hdhalter]

Are we targeting anything for the 2.6 release? If so, does it require documentation? Please create a doc issue in the doc repo. Thanks!

[bharath-techie]

Are we targeting anything for the 2.6 release? If so, does it require documentation? Please create a doc issue in the doc repo. Thanks!

We are not targeting this for 2.6 as of now. So we'll likely create a doc issue for 2.7 release as suggested.

[ahopp]

@bharath-techie I assume (based on https://github.com/opensearch-project/OpenSearch-Dashboards/issues/2909) we bumped this to a later release? Are we now targeting 2.8 at this time?

[bharath-techie]

Yes this will be picked up in a later release.