[Feature] Unified query language selection in Data Explorer

[shanilpa]

Context

The following issue describes a step towards unification towards a common Data Explorer in OpenSearch Dashboards. Currently (v2.11), in OpenSearch Dashboards there is support for several querying languages including DQL (Dashboards Query Language), Lucene, PPL (Piped Processing Language), and SQL. However, these query languages exist in separate applications with no clear way for users to access them outside of just knowing what is supported where. This results in a poor user experience. This issue existed for data sources which we resolved through an improved data source selector component. Now we seek to do the same for query languages.

The challenge

Part of this solution has been implemented in Observability Logs (v2.11) where users can select PPL, SQL and DQL as a query language from a query language selector - DQL redirects users to Discover. Within Discover users do not have a way to select the other available languages. This issue seeks to address this while improving, and unify the experience in Discover and Observability Logs.

Query picker in Observability Logs 2.11

Proposed solution short term

Discover:

Observability Logs:

The proposed solution places the query language switcher in the same location and with the same look and feel in both applications so users know where to find the functionality and expect consistent actions and behaviors.

Additional details

1. Query language is dependent on the data source selected. Some data sources may not allow for querying with certain languages. As a result the query language picker needs to be dynamic and only show the querying languages supported for the selected data source
2. Short term the query language acts as a redirection mechanism between Discover and Observability Logs applications.
3. In future releases we will seek to break the query language selector and query bar from the composed component that contains the time range selector and execute button. This will allow users more space to query but also modularize the querying component for use elsewhere in the application. More information on the proposed North Star for Data Explorer.

[abbyhu2000]

Thanks @shanilpa!

After i start my initial implementation, I have some different opinions on this proposal:

• On the UI mock, the <QueryLanguageSwitcher> is being switched to the left side, and the saved query button is switched to the right side.
  1. Pros: make the observability page and data explorer page look more consistent(since observability page has the query language selector on the left)
  2. Cons:
     1. <QueryLanguageSwitcher> is sub component <QueryBar> which is a sub component of <TopNav> within the data plugin. <TopNav> is an universal component which is used by multiple plugins, such as the dashboard plugin, discover plugin, visualize plugin. It also contains state management logics for persisting data across refresh and also persisting data when we navigate across different pages.

      <div className={className}>
        {this.props.prepend}
        <EuiOutsideClickDetector onOutsideClick={this.onOutsideClick}>
        ...
        </EuiOutsideClickDetector>

        <QueryLanguageSwitcher
          language={this.props.query.language}
          anchorPosition={this.props.languageSwitcherPopoverAnchorPosition}
          onSelectLanguage={this.onSelectLanguage}
          currentApp$={this.props.currentApp$}
        />
      </div>

Switching the locations will make discover page look inconsistent with the rest of the app(core pages such as dashboard page, visualize page). Current user flow include users navigate across visualize page, dashboard page and discover page and expecting the data from top nav(time filter, time refresh interval, global filter) to be persistent, however, making the switch on the top nav UI for discover page alone make this user flow confusing. Since the visualize, dashboard, and discover page are core pages that will always show, and observability page is an optional add-on plugin page, i think it is not worth it to switch the language selector’s position.

Implementation-wise, previously we have the query selector as a pop over button as part of the query bar; and now we want to change the query selector to a drop down and make it separate from the query bar, and put it in front of the query bar. We completely change the structure of this component, so implementation-wise, even though we can do a if(app === ‘data-explorer’), render this new structure, else() -> render the old way, but since it is so different, it feels pretty weird to me if we implement it this way.

I propose three solutions:

1. Keep the language selector on the right, and making it a dropdown for the discover page. This way can keep the discover page look consistent with the rest of the app while also providing the options to choose PPL as well.
2. The observability page should use the universal <TopNav> component that is provided by dashboard core, so it looks persistent with the rest of the core app.
3. Detach the sub-components of <TopNav> and redesign so it provides more flexibility for the pages to arrange the components. This is a much bigger scope task since <TopNav> component involves with lots of the state managements logics.

---

• In the proposal, it also mentions that when we do not have observability plugin installed, we should keep the <TopNav> for discover page as it is before. This requires that we add the observability plugin as an optional plugin dependency for the data plugin. However, when i try to implement it, there is an architectural conflicts — observability plugin has data plugin as a required dependency, and data plugin has observability plugin as an optional dependency while also being required to figure out whether observability plugin exists or not by itself. When i was implementing it this way, the compiler is complaining on this circular dependency. One solution is that: Observability plugin could add some service so that whenever it is installed, it will release some kind of signals to let the core plugin know that it installs.

Do you guys have any thoughts on this? @shanilpa @joshuarrrr @kavilla @AMoo-Miki

[shanilpa]

Thanks for the comment and deep dive into the implementation details @abbyhu2000.

I understand the need to make the experience consistent across the application however, the needs of the Discover application are evolving as we think about it's evolution into Data Explorer. These needs do not align with the needs of the < top nav > in other applications such as Dashboards anymore. For example, I don't see a clear customer need for a Dashboard to allow users to switch query languages.

Currently, the top nav used in the core application (Discover, Visualize Dashboards) isn't used in Observability. I am assuming the reason for this is due to the limitations of this component. If we look at the query bar for DQL - that is simply search which from a UX side can be handled in a single line for most cases. However, a query language like PPL or SQL requires wrapping of text, in this case that top nav has to be broken to support the needs of the query language - probably why Observability can't use the core component.

As we try and bring our experiences together and Data Explorer seeks to make things more cohesive for users we will have to take a look at redefining the top nav to support the unique needs of Data Explorer. Breaking up this component, does not mean we lose consistency across the application, but rather an opportunity to define something more flexible and scaleable that can deliver better experiences across the board.

With this said, I understand there are technical limitations with relation to the data plugin and how tightly the front end component is tied to it. I'd love to know more and see how we can evolve both towards a better experience for customers, and builders looking to use our core components in other places.

cc @ahopp @dagneyb if you have any thoughts and comments.

[ashwin-pc]

@shanilpa There are two different problems we are trying to solve here and would love to hear your thoughts on my suggested approaches to solving for this.

Problem 1: Having more language options than just DQL and Lucene in Data Explorer (Needs to be dynamic such that plugins can define what they are) and switching to a view that can support them when selected.

Problem 2: Displaying it in a way thats in line with the north star

I have a few suggestions on how to solve them both with or without any modifications to the proposal, each with their pros and cons.

Solution 1: Create a new querybar component for Data Explorer.

Pros:

• This approach no longer uses the standard query bar component from the data plugin and instead creates a new one that can evolve over time into the north star query component.
• It also has the least tech debt because it does not add any unnecessary logic to the existing data plugin component
• It can be implemented similar to the north star using the context bar for the time picker and the canvas for the query bar

Cons:

• Initial investment to recreate the existing query bar functionality without breaking any features.

Solution 2: Modify the existing querybar to work differently in Data explorer

Pros:

• Quicker to implement
• Less duplication of code

Cons:

• More tech debt because the north star querybar will have to be implemented quite differently anyways
• Hardcoding. Since this is a short term solution, it does not make sense to create a generalized solution to solve the problem of having more query languages in the shared querybar component. So the only option here will be to hard code the extra logic to add PPL if the observability plugin exists. This is dangerous if left in there long term since historical context can be lost over time and break things. This applies to both the look and feel and the query selection problem too.

If there is no immediate urgency to implement the short term solution (and by immediate i mean you need it in a week or two), i'd highly recommend the first solution since that sets us up better for success with data explorer long term and also front loads a lot of the work we will have to do next to get to the north star.

[shanilpa]

Thanks for the breakdown @ashwin-pc I'm aligned with Solution 1 from a UX perspective. I think it puts us in a better position to deliver more focused, scaleable and useful query functionality to users.

@dagneyb, @ahopp thoughts?

[dagneyb]

I'm aligned with Solution 1, but am interested in the LoE difference between the two approaches. My answer changes if its 2 weeks of work vs. 2 years :). @ashwin-pc could you do a high level estimate?

[shanilpa]

Updating with the most recent mocks for Discover and placement of the query language selector

[kavilla]

Work related to this: https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6613