wazuh-dashboard-plugins icon indicating copy to clipboard operation
wazuh-dashboard-plugins copied to clipboard

Published 20 hours ago • verified publisher icon wazuh

New search bar component

Open AlexRuiz7 opened this issue 2 years ago • 1 comments

Summary

Our current SearchBar component is far from working properly, and it's being harder and harder to maintain over time.

We need to deprecate this component in favor of a better one, and use it consistently along the App.

This Epic will hold the issues of the current component to be used as functional requirements for the new component, and also will cover its design and development.

Functional requirements

  1. The search bar must allow any operator defined by the Wazuh Query Language, including parentheses. #4313
  2. The search bar must allow values to contain and and or on their names, including the white spaces. #4088

Non-functional requirements

To be done.

Related issues

  • #4313
  • #4088
  • #3968
  • #3818
  • #3325

Design

There are several designs on the table at the moment. The most supported one is to use the Wazuh Query Language as is, with no translations, as was done on the previous search bar, which translated any and and or in the query with ; and ,, respectively.

AlexRuiz7 avatar Jul 06 '22 15:07 AlexRuiz7

Research about current SearchBar behavior

sequenceDiagram
WzSearchBar->>SuggestHandler: input query in string
SuggestHandler ->> QInterpreter: query string
loop
    QInterpreter->>QInterpreter: PARSE and VALIDATE the query
end
QInterpreter->>SuggestHandler: response is VALID or INVALID
SuggestHandler->>WzSearchBadges: send VALID filters
Note right of WzSearchBadges: Add badges for each VALID filter

Query Filters route

graph TD;
UISearchBar-->ValidFiltersList-->filtersToObject-->WazuhManager(API)

Machi3mfl avatar Jul 15 '22 14:07 Machi3mfl

The current search bar translates the input to the expected Wazuh query syntax. This has some problems related to the and and or words, that translate to AND (;) and OR (,) operators of Wazuh query syntax. This affects to manage of when should suggest a new field.

I think the easier approach is using directly the Wazuh query language without translations. The problem with this approach is that is more complicated for users that don't know the query language.

Idea

I think we could offer multiple query languages.

One of them, should be the Wazuh query language without translations, this would avoid any problems on our side.

Moreover, we could add a user-friendly language/interface to the query that could be based on the current query syntax provided by the WzSearchBar component. It will have the same difficulties as the current query syntax.

This approach is based on the search bar used to query to Elasticseach/Wazuh indexer in the Discover or Dashboard plugins.

Image

Desvelao avatar Feb 09 '23 12:02 Desvelao

POC - Search bar - Multiple query languages

I am working around the idea of a search bar that supports multiple query languages. The selection can be changed through a control.

This base component expects the query language to be registered and has a specific interface to interact with.

The base component only manages the input text, then this is passed to the selected query language to analyze the query and returns the properties that should use the base component. This enables to the extension of the functionality.

The new search bar base component is based on a custom EuiSuggest component (the current WzSearchBar wraps it).

I tried to use the EuiSuggest component from the @elastic/eui, but it seems to have some bugs. I guess this is the reason because the EuiSuggest component was recreated as a custom one in the plugin source code to use in WzSearchBar component.

The approach of this POC, let to decouple the business logic of each query language. For example, adding suggestions according to the input query or defining the action when a suggestion item is clicked.

Desvelao avatar Feb 14 '23 11:02 Desvelao

POC - Query language - API query language

Taking as the base the POC of the new search bar component that supports multiple query languages, I am working on a compatible interface that is based on a similar approach to the Wazuh API query language https://documentation.wazuh.com/current/user-manual/api/queries.html.

The behavior is similar to the current WzSearchBar. This means displaying suggestions to help the user.

The current implementation analyzes (through a custom analysis) the query and tokenizes it. Then, these tokens are used to get the suggestions that will be displayed in the search bar.

I have added some tests with use cases to test the tokenizer and the suggestions handler.

Features:

  • Get suggestions from the input
  • Supports dynamic suggestions (let to build the suggestions from any source that requires fetching data)

This query language has not user-friendly or easily understandable syntax by not familiar users. The purpose of adding this query language is to prevent the interpretation of a high-level query syntax as the current WzSearchBar component that causes some queries don't work as expected. The idea of the new search bar that supports multiple query languages is to add a new high-level syntax more user-friendly and allows the user to choose which query language to use.

Screenshots:

image

Open the query language information

This contains information about the current query language and a selector to change the selected query language.

image

Suggestions

image image image image

Desvelao avatar Feb 14 '23 12:02 Desvelao

@wazuh/framework will complete an investigation on the reports done by @Desvelao and our current implementation of the q parameter.

davidjiglesias avatar Feb 14 '23 13:02 davidjiglesias

To prevent some differences between the expected queries by the Wazuh API and the implementation of the query language in the search bar, we must know the implementation and decomposition of the query through the q query parameter used in Wazuh API endpoints. This knowledge could help us to replicate the business logic and reduce the discordances.

Desvelao avatar Feb 14 '23 13:02 Desvelao

POC - Search bar - Query Language custom behavior

I am working around the idea to allow each query language can define custom behaviors.

As a POC, I added a prepend element to the search bar to indicate that there is a implicit filter. This is useful to display that there is a non-editable query that will be added to the user query. This case is in the search bar located in the Agents section, that negates the data related to the Wazuh managers (id is not 000).

Screenshots

Image

I need to refactor some logic in the POC of new search bar component.

Desvelao avatar Feb 15 '23 08:02 Desvelao

POC - Search bar - Multiple query languages

I uploaded a POC of this search bar to the branch https://github.com/wazuh/wazuh-kibana-app/tree/feat/4312-poc-search-bar-component-multiple-query-language-api-query-language-implementation.

This POC contains:

  • New search bar component that supports multiple query languages
  • Query languages:
    • AQL: custom implementation of a similar approach of Wazuh Query Language
      • Include suggestions
    • UIQL: simple implementation
  • The search bar was implemented as an example in the section of Agents

Desvelao avatar Feb 16 '23 10:02 Desvelao

The work in the POC will be on hold until the @wazuh/framework can give us details about the implementation and analysis of the query in the q query parameter. This knowledge will let us work on the implementation of the query language and provide suggestions to the user.

Desvelao avatar Feb 16 '23 12:02 Desvelao

Discussion

In today's meeting, it was decided the framework team will work to provide knowledge how the query decomposition works in the q query parameter.

Desvelao avatar Feb 23 '23 11:02 Desvelao

Decomposition of q API parameter

The following is the procedure applied in the framework for those requests that include q. This is a somewhat complex process with multiple particularities depending on the data to be obtained. The explanation will be based on this example request:

GET /agents?q=(status=active;id<3),(status=disconnected);(name~c)

1. Regex

First, it is checked if the value of q matches with the following regex:

        # To correctly turn a query into SQL, a regex is used. This regex will extract all necessary information:
        # For example, the following regex -> (name!=wazuh;id>5),group=webserver <- would return 3 different matches:
        #   (name != wazuh ;
        #    id   > 5      ),
        #    group=webserver
        self.query_regex = re.compile(
            # A ( character.
            r"(\()?" +
            # Field name: name of the field to look on DB.
            r"([\w.]+)" +
            # Operator: looks for '=', '!=', '<', '>' or '~'.
            rf"([{''.join(self.query_operators.keys())}]{{1,2}})" +
            # Value: A string.
            r"((?:(?:\((?:\[[\[\]\w _\-.,:?\\/'\"=@%<>{}]*]|[\[\]\w _\-.:?\\/'\"=@%<>{}]*)\))*"
            r"(?:\[[\[\]\w _\-.,:?\\/'\"=@%<>{}]*]|[\[\]\w _\-.:?\\/'\"=@%<>{}]+)"
            r"(?:\((?:\[[\[\]\w _\-.,:?\\/'\"=@%<>{}]*]|[\[\]\w _\-.:?\\/'\"=@%<>{}]*)\))*)+)" +
            # A ) character.
            r"(\))?" +
            # Separator: looks for ';', ',' or nothing.
            rf"([{''.join(self.query_separators.keys())}])?"
        )

(\()?([\w.]+)([=!=<>~]{1,2})((?:(?:\((?:\[[\[\]\w _\-.,:?\\/'\"=@%<>{}]*]|[\[\]\w _\-.:?\\/'\"=@%<>{}]*)\))*(?:\[[\[\]\w _\-.,:?\\/'\"=@%<>{}]*]|[\[\]\w _\-.:?\\/'\"=@%<>{}]+)(?:\((?:\[[\[\]\w _\-.,:?\\/'\"=@%<>{}]*]|[\[\]\w _\-.:?\\/'\"=@%<>{}]*)\))*)+)(\))?([,;])?

Here its parts can be better visualized: image

It has undergone many changes, and its current state is the result of this issue:

  • https://github.com/wazuh/wazuh/issues/13369

2. Extract matching groups

From the original query ((status=active;id<3),(status=disconnected);(name~c)) tuples are extracted with 6 groups for each of them (1. opening parenthesis, 2. field name, 3. operator, 4. value, 5. closing parenthesis, 6. separator) (link):

[
    ('(', 'status', '=', 'active', '', ';'), 
    ('', 'id', '<', '3', ')', ','), 
    ('(', 'status', '=', 'disconnected', ')', ';'), 
    ('(', 'name', '~', 'c', ')', '')
]

If any of the field names (2.) do not match those predefined for the resource type (in this case, agent), an error message is raised. The same happens with operators: https://github.com/wazuh/wazuh/blob/36d17b36a732ecc4b4bffda7ddd047d60a3c79ac/framework/wazuh/core/utils.py#L1505-L1506

3. Process matching groups

Each tuple is iterated, processing each field name and value if necessary. For example:

  • The characters " and ' are replaced by wildcards (_) (link).
  • A subset of valid operators is used if the field name is group. Wildcards are also included. This is an example of specific processing for agent queries. (link).
  • If it includes operations with date type data, they are transformed into the appropriate format according to the operation (link).

Result

After applying the above steps (and a few more that are not specific to the q parameter), the example query (status=active;id<3),(status=disconnected);(name~c) is transformed into this part of a SQL statement:

((connection_status = 'active' COLLATE NOCASE) AND (id < '3' COLLATE NOCASE)) OR (connection_status = 'disconnected' COLLATE NOCASE) AND (name LIKE '%c%' COLLATE NOCASE)

The complete SQL query that would be sent to wazuh-db for this GET /agents request would be this:

SELECT disconnection_time as 'disconnection_time',os_arch as 'os.arch',os_major as 'os.major',config_sum as 'configSum',last_keepalive as 'lastKeepAlive',os_minor as 'os.minor',connection_status as 'status',os_uname as 'os.uname',coalesce(ip,register_ip) as 'ip',os_build as 'os.build',os_version as 'os.version',internal_key as 'internal_key',id as 'id',date_add as 'dateAdd',`group` as 'group',os_codename as 'os.codename',name as 'name',merged_sum as 'mergedSum',version as 'version',node_name as 'node_name',register_ip as 'registerIP',os_platform as 'os.platform',group_config_status as 'group_config_status',os_name as 'os.name',manager_host as 'manager' FROM agent WHERE ((connection_status = 'active' COLLATE NOCASE) AND (id < '3' COLLATE NOCASE)) OR (connection_status = 'disconnected' COLLATE NOCASE) AND (name LIKE '%c%' COLLATE NOCASE)   ORDER BY id ASC LIMIT 500 OFFSET 0

Selutario avatar Feb 23 '23 16:02 Selutario

Thank you so much for the q parameter analysis @Selutario!

I will work to enhance the current API query language implementation in the search bar.

Desvelao avatar Feb 24 '23 10:02 Desvelao

I would like to implement a validation of the query in the search bar to avoid or reduce and indicate to the user the query is invalid.

@Selutario, is there some validation that the API is checking according to the schema? or validating if the group operators (opening and closing) are in the correct place or grouping level? Any more consideration? If yes, could you provide the information?

Thank you so much!

Desvelao avatar Feb 27 '23 09:02 Desvelao

POC - Query language - API query language

In a previous meeting related to the search query done in https://github.com/wazuh/wazuh-kibana-app/pull/5196, it was mentioned that the new search bar and AQL implementation could use this query when the user introduces a search term, searching in each used (table columns) through a like (~).

The current implementation of AQL doesn't support this functionality so the user should introduce the query manually.

We should discuss if this feature should be allowed by the implementation or not.

Desvelao avatar Feb 27 '23 09:02 Desvelao

POC - Query language - API query language - Hidding the suggestions when using Search suggestion.

The search bar displays a suggestion to run the search query. When clicking it, the suggestions popover disappeared for a short period of time and this appeared again.

image

Digging

I was researching the cause of this. I reviewed the current WzSearchBar implementation that changes the visibility of the suggestions popover and blurs the input. This seems not to work as expected, and the suggestion popover appeared after using the Search suggestion, the same for the Apply filter suggestion of the WzSearchBar.

Clicking in a suggestion item of the popover, ran a handler. This handler called the onSearch function and close the suggestion popover and blurred the input (same logic as the current WzSearchBar), then a focus event was triggered causing the visibility of the suggestion popover is on. I didn't understand the reason for this behavior so I researched the cause of this.

I researched and tried multiple things without success.

Finally, I asked for help and I had a meeting with @asteriscos, where I was explaining him the problem to review if I was missing something. We concluded that something unknown and not intentional by our side is firing the focus event in the input. So I researched the components used. One of them is coming from the opensearch-project/oui that is called EuiInputPopover https://github.com/opensearch-project/oui/blob/1.0.0/src/components/popover/input_popover.ts. The EuiInputPopover uses a focus trap, which could be the cause of the focus event being fired. I found a property to disable this behavior called disableFocusTrap https://github.com/opensearch-project/oui/blob/1.0.0/src/components/popover/input_popover.tsx#L141. Adding the disableFocusTrap property to the EuiInputPopover fixed the problem of the undesired behavior.

Desvelao avatar Mar 02 '23 15:03 Desvelao

POC - Query language - API query language

Fixes

  • Fixed a minor problem related to input text with undefined value

Desvelao avatar Mar 02 '23 15:03 Desvelao

Answering these questions:

Is there some validation that the API is checking according to the schema?

Yes, there are multiple validations in addition to the regex. For each match (<opening_parenthesis><field><operator><value><closing_parenthesis><separator>) within the query, the following validations are done:

1. Check whether field is allowed

Code:

            if field not in self.fields.keys():
                raise WazuhError(1408, "Available fields: {}. Field: {}".format(', '.join(self.fields), field))

The allowed fields vary for each type of data, not being the same for agents as for syscollector (for example). The following is a list of all allowed fields for each of the endpoints that access wazuh-db:

In addition, those marked with an asterisk do not get the data from the database. This means that they do not follow the process mentioned here. Therefore, although in those cases the q parameter tries to mimic the behavior of the other listed endpoints, its behavior is not identical (for example, it does not verify that the fields used are allowed, simply nothing is returned if the field does not exist).

Is validated whether group operators (opening and closing) are in the correct place or grouping level?

No, it is not validated. However, wazuh-db will return an error response if those groups are wrong.

Any more consideration?

WIP

Selutario avatar Mar 02 '23 16:03 Selutario

Answering these questions:

Is there some validation that the API is checking according to the schema?

Yes, there are multiple validations in addition to the regex. For each match (<opening_parenthesis><field><operator><value><closing_parenthesis><separator>) within the query, the following validations are done:

1. Check whether field is allowed

Code:

            if field not in self.fields.keys():
                raise WazuhError(1408, "Available fields: {}. Field: {}".format(', '.join(self.fields), field))

The allowed fields vary for each type of data, not being the same for agents as for syscollector (for example). The following is a list of all allowed fields for each of the endpoints that access wazuh-db:

In addition, those marked with an asterisk do not get the data from the database. This means that they do not follow the process mentioned here. Therefore, although in those cases the q parameter tries to mimic the behavior of the other listed endpoints, its behavior is not identical (for example, it does not verify that the fields used are allowed, simply nothing is returned if the field does not exist).

Is validated whether group operators (opening and closing) are in the correct place or grouping level?

No, it is not validated. However, wazuh-db will return an error response if those groups are wrong.

Any more consideration?

WIP

This is useful to add minimal validation in the search bar and inform to the user of a problem with the input query.

I will see to add the validation of field to the POC. We thought about what to do with the validation of group operators.

At the moment, I have no more considerations.

Thank you so much for the information @Selutario!

Desvelao avatar Mar 03 '23 07:03 Desvelao

POC - Search bar - Display input query error

The new search bar component wraps the custom EuiSuggest component. If we want to be able to display a message about the input validation error, we will have to modify the custom EuiSuggestInput component (public/components/eui-suggest/suggest_input.js) wrapping the EuiFieldText with a EuiFormRow that enables to display the error message.

For the required query language implementations, they should validate the user input and returns the validation error when the input analysis is done. This property could be forwarded to the EuiFormRow component to display the error and inform to the user.

The validation of the input query could be useful to prevent the user can run an invalid query and inform that there is an error.

Desvelao avatar Mar 03 '23 08:03 Desvelao

POC - Query language - API query language

In a previous meeting related to the search query done in #5196, it was mentioned that the new search bar and AQL implementation could use this query when the user introduces a search term, searching in each used (table columns) through a like (~).

The current implementation of AQL doesn't support this functionality so the user should introduce the query manually.

We should discuss if this feature should be allowed by the implementation or not.

Some options:

  • Manage from the user input. We should differentiate if the input is a valid query for q or is a query that should be searched in each field using the like operator ~.

User-friendly Complicated implementation and could cause unexpected behaviors.

  • When there is some input, display a suggestion that when clicked, runs the search transforming the query to use the like operator in all the expected fields.

Less user-friendly than the previous option Less complicated of implementing

As I said in the quoted message, we should discuss if we want to implement this feature.

Desvelao avatar Mar 03 '23 09:03 Desvelao

POC - Search bar

Changes

  • Add to the search bar example implementation the ability to receive filters from the external components to the search bar
  • Enhance the search bar component documentation

Desvelao avatar Mar 03 '23 14:03 Desvelao

Disccussion

In today's meeting, I was exposing the work done on this issue and the POC.

The action items to do are:

  • Implement a new query language, similar to the current one of WzSearchBar that solves the problems of this.

The query language can't be easily ported to the new search bar component.

Desvelao avatar Mar 06 '23 11:03 Desvelao

POC - Search bar - HAQL

I was working on a high-level implementation based on the Wazuh API Query Language.

The syntax is similar to the current query language used by the current WzSearchBar component. It uses a human-understandable logical operators (and, or) instead of the used by the Wazuh API Query Langue.

HAQL is the key name of this language. This could be changed. It means Human API Query Language.

I added the ability to define the values wrapped with quotes " and the quote " can be escaped using a \ escape character (\").

Moreover, the usage of whitespace to separate entities is optional.

Features:

  • Display suggestions
  • Support implicit query
  • Support external query

I added documentation related to this query language.

I added tests to test some functionalities.

Screenshots image image image image image image image image image image

Desvelao avatar Mar 07 '23 16:03 Desvelao

POC - Search bar - Multiple query languages

If we finally implement the search bar supporting multiple languages, we should add a mechanism to persist the selection of query language. This could be done preferably through local storage.

  • [ ] Add a mechanism to persist the selected query language

Desvelao avatar Mar 08 '23 07:03 Desvelao

Checkpoint

Show:

  • HAQL query language

Review:

  • Issues linked to this issue in the initial comment

Next things need to be discussed:

  • Ability to input a search term (require to define the query done to the Wazuh API). Apply to HAQL and AQL?
  • Will multiple languages be supported? Then, add a mechanism to remember the selected language. https://github.com/wazuh/wazuh-kibana-app/issues/4312#issuecomment-1459649182
  • Final query language names
  • Next steps
    • Implement the new search bar
      • Replace usage of the current search bar
      • Remove the current search bar definition

Desvelao avatar Mar 14 '23 11:03 Desvelao

Meeting

We had a meeting to see the checkpoint https://github.com/wazuh/wazuh-kibana-app/issues/4312#issuecomment-1467937462

Action items:

  • [x] Only use the high-level API query language (HAQL)
  • [x] Rename HAQL to WQL (Wazuh Query Language)
  • [x] Add tests with operators within the value. They should be supported. Found some problems https://github.com/wazuh/wazuh-kibana-app/issues/4312#issuecomment-1469956982.
  • [x] Implement the ability to search terms through an API query as: fied1~value;field2~value. Use the current differentiation of WzSearchBar (operator doesn't exist) to decide if it is a search or is using the query language syntax.

Desvelao avatar Mar 15 '23 11:03 Desvelao

Changes

  • Renamed HAQL to WQL (Wazuh Query Language)

Desvelao avatar Mar 15 '23 12:03 Desvelao

POC - WQL - Use cases

I found some use cases where some characters are not supported to be in the value token:

  • !
  • ~

Using the tokenizer of AQL (implementation of the regular expression used in API), the mentioned characters are tokenized as operators instead of forming part of the value.

Details

Queries in AQL.

  • field=with != value

Decomposition field: field operator: = value: with (note the whitespace) field: missing operator: != value: value (note the whitespace)

  • field=with ~ value

Decomposition field: field operator: = value: with (note the whitespace) field: missing operator: ~ value: value (note the whitespace)

Both characters can't be part of the value according to the API regular expression mentioned here: https://github.com/wazuh/wazuh-kibana-app/issues/4312#issuecomment-1442091645

This could cause a problem if using suggestions or user input that contains these characters in the values. The API request could fail or the API misinterpret the query.

Desvelao avatar Mar 15 '23 12:03 Desvelao

Changes

  • WQL

    • Add more use cases to the test of WQL query language
    • Replace some literals with constants

  • Enhance documentation of query languages

  • Added a title to the popover of the syntax options image

  • Wrapped the user input with group operators () when there is a implicit query. Done for AQL and WQL query languages.

Desvelao avatar Mar 15 '23 13:03 Desvelao

POC - WQL - Implicit query term

I was working to add the ability to allow to the user types a query term and under the hoods, this is translated to the expected query.

The translation is based in the compare the query term with each available field using the ~ (like as) operator. If the user types:

linux

If the available fields are id and ip, this query term is translated to am unified query as:

id~linux,ip~linux

This feature tries to replace the feature of the current search that is using the search query parameter of the Wazuh API endpoints. The difference is the search query parameter searches in each field of the item, and the suggested approach, only searches in the specified fields. This allows the search to be more exact to the use cases where there fields that are not displayed in the UI. (https://github.com/wazuh/wazuh-kibana-app/issues/3968#issuecomment-1420627790)

image

Desvelao avatar Mar 16 '23 15:03 Desvelao