Improve the Wazuh Indexer ISM policies E2E test with test validation

[rauldpm]

Description

Reviewing the https://github.com/wazuh/wazuh/issues/25828 issue, I noticed that the steps to be done are executed correctly most of the time, but we are not validating those changes, we should modify the E2E test to validate the changes and check that the retention policy works as expected

This issue also expects to deploy a Wazuh agent on Red Hat 8 and Windows 11, so it would make sense to test the policy retention with data provided by those agents, if not, the agent deployment should be removed

[rauldpm]

We need to discuss if we want to include this in RC 2 or not, as operational the issue can miss the release version, although it is desired to complete it before starting a release testing

I propose to set 4.10.0 Alpha 2 as version target instead

[joaquinsgi]

Two agents (Red Hat 8 and Windows 11) will be deployed, along with one manager, to validate those steps and check if the retention policy works properly.

[joaquinsgi]

I follow the steps in:

• https://github.com/wazuh/wazuh/issues/25828

and

• https://documentation-dev.wazuh.com/v4.9.1-rc2/user-manual/wazuh-indexer/index-life-management.html#using-the-visual-editor

To create a new retention policy, there are two different ways:

1. Using the Visual editor
2. Using the JSON editor

Following Using the Visual editor

Firstly, I followed the steps from the video regarding the previously commented issue. And all worked correctly, without any problems.

Secondly, I followed the steps in the documentation, and I noticed that there is a lack of images in the documentation from step number 3, maybe we should add more images to improve the understanding of the user.

Following Using the JSON editor Firstly I tried to follow the video from the issue, but I didn't have the JSON, so I had to use the JSON from the documentation, a good point is that if you want to create one policy with the visual editor and another one with the JSON editor and you use in both options the same index patterns, It appears a warning message telling you to change the priority from one of them:

Secondly, the Documentation it's very well achieved using the JSON editor.

The last step is to validate those retention policies: I followed the following comment to validate those retentions:

• https://github.com/wazuh/wazuh/issues/25828#issuecomment-2372541407

So I applied the policy into wazuh-alerts-4.x-2024.10.04, following the issue

Firstly I checked the wazuh-alerts-4.x-2024.10.04 before I applied the policy.

After a couple of minutes. It can be seen that the size has decreased:

So I can validate that the retention policy works normally.

But some points need to be clarified:

• We should add more images to improve the understanding of the user in the documentation form step number 3.
• In the tests, if we create a policy for wazuh-alerts, we should validate that retention policy, not for wazuh-archives.
• To create a retention policy from the JSON editor, that JSON has to be written in the comment.
• In the documentation, if we create a retention policy for wazuh-alerts, the applying has to be using that policy not for wazuh-archives

[hossam1522]

LGTM!

[juliamagan]

The default documentation proposes 90d, we should add a test with a shorter time to be able to check that the changes are really applied and the alerts change storage.

[joaquinsgi]

I suggest that we should follow the next template for the following tests:

End-to-End (E2E) Testing Guideline

• Documentation: Always consult the development documentation for the current stage tag at this link. Be careful because some of the description steps might refer to a current version in production, always navigate using the current development documention for the stage under test. Also, visit the following pre-release package guide to understand how to modify certain links and urls for the correct testing of the development packages.
• Test Requirements: Ensure your test comprehensively includes a full stack and agent/s deployment as per the Deployment requirements, detailing the machine OS, installed version, and revision.
• Deployment Options: While deployments can be local (using VMs, Vagrant, etc) or on the aws-dev account, opt for local deployments when feasible. For AWS access, coordinate with the DevOps team through this link.
• External Accounts: If tests require third-party accounts (e.g., GitHub, Azure, AWS, GCP), request the necessary access through the DevOps team here.
• Alerts: Every test should generate a minimum of one end-to-end alert, from the agent to the dashboard, irrespective of test type.
• Multi-node Testing: For multi-node wazuh-manager tests, ensure agents are connected to both workers and the master node.
• Package Verification: Use the pre-release package that matches the current TAG you're testing. Confirm its version and revision.
• Filebeat Errors: If you encounter errors with Filebeat during testing, refer to this Slack discussion for insights and resolutions.
• Known Issues: Familiarize yourself with previously reported issues in the Known Issues section. This helps in identifying already recognized errors during testing.
• Reporting New Issues: Any new errors discovered during testing that aren't listed under Known Issues should be reported. Assign the issue to the corresponding team (QA if unsure), add the Release testing objective and Urgent priority. Communicate these to the team and QA via the c-release Slack channel.
• Test Conduct: It's imperative to be thorough in your testing, offering enough detail for reviewers. Incomplete tests might necessitate a redo.
• Documentation Feedback: Encountering documentation gaps, unclear guidelines, or anything that disrupts the testing or UX? Open an issue, especially if it's not listed under Known Issues. Please answer the feedback section, this is a mandatory step.
• Format: If this is your first time doing this, refer to the format (but not necessarily the content, as it may vary) of previous E2E tests, here you have an example https://github.com/wazuh/wazuh/issues/13994.
• Status and completion: Change the issue status within your team project accordingly. Once you finish testing and write the conclusions, move it to Pending review and notify the @wazuh/devel-indexer team via Slack using the c-release channel. Beware that the reviewers might request additional information or task repetitions.
• For reviewers: Please move the issue to Pending final review and notify via Slack using the same thread if everything is ok, otherwise, perform an issue update with the requested changes and move it to On hold, increase the review_cycles in the team project by one and notify the issue assignee via Slack using the same thread.

For the conclusions and the issue testing and updates, use the following legend:

Status legend

• 🟢 All checks passed
• 🟡 Found a known issue
• 🔴 Found a new error

Issue delivery and completion

• Initial delivery: The issue's assignee must complete the testing and deliver the results by Sep 23, 2024 and notify the @wazuh/devel-indexer team via Slack using the c-release channel
• Review: The @wazuh/devel-indexer team will assign a reviewer and add it to the review_assignee field in the project. The reviewer must then review the test steps and results. Ensure that all iteration cycles are completed by Sep 24, 2024 date (issue must be in Pending final review status) and notify the QA team via Slack using the c-release channel.
• Auditor: The QA team must audit, validate the results, and close the issue by Sep 25, 2024.

Deployment requirements

Component	Installation	Type	OS
Indexer	Quickstart	-	Red Hat Enterprise Linux 8
Server	Same as indexer, all-in-one	-	-
Dashboard	Same as indexer, all-in-one	-	-
Agent	Installing Wazuh agents	-	Red Hat Enterprise Linux 8 x86_64, Windows 11 x86_64

Test description

0. Follow and read documentation links to test ISM policies in Wazuh Indexer:

https://documentation-dev.wazuh.com/v4.9.1-rc1/user-manual/wazuh-indexer/index-life-management.html https://opensearch.org/docs/latest/im-plugin/ism/index/

1. Create a retention policy using visual editor (5m)

2. Create a retention policy using json editor (5m)

3. Applying the retention policy to alerts index (5m)

4. Validate that retention policy, checking the size from the file

5. Wazuh agent installation

Known issues

There are no known issues.

Conclusions

Summarize the errors detected (Known Issues included). Illustrate using the table below. REMOVE CURRENT EXAMPLES:

Status	Test	Failure type	Notes
:black_circle:	Creating a retention policy using visual editor
:black_circle:	Creating a retention policy using json editor
:black_circle:	Applying the retention policy to alerts index
:black_circle:	Verify that the retention policy worked
:black_circle:	Wazuh agent installation
:black_circle:	Roll Over

Feedback

We value your feedback. Please provide insights on your testing experience.

• Was the testing guideline clear? Were there any ambiguities?
  • Yes steps were clear and easy to follow. nonetheless lack of instructions regardin how to perform Roll over + Alias, what exactly is expected to be done and what should be the outcome of the test.
• Did you face any challenges not covered by the guideline?
  • **Yes, the guidelines do not contain verification steps to ensure the retention policy does work. Also no mention on how to perform rollback in the documentation. **
• Suggestions for improvement:
  • Create documentation on how to create and manage data streams and performing roll over. See Issue#5771
  • When creating a retention policy, set it to 5 minutes, and in the step of verifying whether the retention policy worked, check if the size of the files decreased.
  • Perform the step of verifying the retention policy after applying that policy.

Reviewers validation

The criteria for completing this task is based on the validation of the conclusions and the test results by all reviewers.

All the checkboxes below must be marked in order to close this issue.

• [ ] @juliamagan
• [ ] @wazuh/devel-indexer

[juliamagan]

Why is the agent installation useful? We should add a step that generates an alert, and then check the rotation. In addition, the steps guide us to apply the same retention twice, in different ways, but to check them once later. We should check that the policy is applied every time we modify it, and there should be some difference between the two policies so we are sure that they are being applied and the system is not using the previous one.

[joaquinsgi]

Thanks @juliamagan im totally agree with you, so I suggest the test should be like the following:

End-to-End (E2E) Testing Guideline

• Documentation: Always consult the development documentation for the current stage tag at this link. Be careful because some of the description steps might refer to a current version in production, always navigate using the current development documention for the stage under test. Also, visit the following pre-release package guide to understand how to modify certain links and urls for the correct testing of the development packages.
• Test Requirements: Ensure your test comprehensively includes a full stack and agent/s deployment as per the Deployment requirements, detailing the machine OS, installed version, and revision.
• Deployment Options: While deployments can be local (using VMs, Vagrant, etc) or on the aws-dev account, opt for local deployments when feasible. For AWS access, coordinate with the DevOps team through this link.
• External Accounts: If tests require third-party accounts (e.g., GitHub, Azure, AWS, GCP), request the necessary access through the DevOps team here.
• Alerts: Every test should generate a minimum of one end-to-end alert, from the agent to the dashboard, irrespective of test type.
• Multi-node Testing: For multi-node wazuh-manager tests, ensure agents are connected to both workers and the master node.
• Package Verification: Use the pre-release package that matches the current TAG you're testing. Confirm its version and revision.
• Filebeat Errors: If you encounter errors with Filebeat during testing, refer to this Slack discussion for insights and resolutions.
• Known Issues: Familiarize yourself with previously reported issues in the Known Issues section. This helps in identifying already recognized errors during testing.
• Reporting New Issues: Any new errors discovered during testing that aren't listed under Known Issues should be reported. Assign the issue to the corresponding team (QA if unsure), add the Release testing objective and Urgent priority. Communicate these to the team and QA via the c-release Slack channel.
• Test Conduct: It's imperative to be thorough in your testing, offering enough detail for reviewers. Incomplete tests might necessitate a redo.
• Documentation Feedback: Encountering documentation gaps, unclear guidelines, or anything that disrupts the testing or UX? Open an issue, especially if it's not listed under Known Issues. Please answer the feedback section, this is a mandatory step.
• Format: If this is your first time doing this, refer to the format (but not necessarily the content, as it may vary) of previous E2E tests, here you have an example https://github.com/wazuh/wazuh/issues/13994.
• Status and completion: Change the issue status within your team project accordingly. Once you finish testing and write the conclusions, move it to Pending review and notify the @wazuh/devel-indexer team via Slack using the c-release channel. Beware that the reviewers might request additional information or task repetitions.
• For reviewers: Please move the issue to Pending final review and notify via Slack using the same thread if everything is ok, otherwise, perform an issue update with the requested changes and move it to On hold, increase the review_cycles in the team project by one and notify the issue assignee via Slack using the same thread.

For the conclusions and the issue testing and updates, use the following legend:

Status legend

• 🟢 All checks passed
• 🟡 Found a known issue
• 🔴 Found a new error

Issue delivery and completion

• Initial delivery: The issue's assignee must complete the testing and deliver the results by Sep 23, 2024 and notify the @wazuh/devel-indexer team via Slack using the c-release channel
• Review: The @wazuh/devel-indexer team will assign a reviewer and add it to the review_assignee field in the project. The reviewer must then review the test steps and results. Ensure that all iteration cycles are completed by Sep 24, 2024 date (issue must be in Pending final review status) and notify the QA team via Slack using the c-release channel.
• Auditor: The QA team must audit, validate the results, and close the issue by Sep 25, 2024.

Deployment requirements

Component	Installation	Type	OS
Indexer	Quickstart	-	Red Hat Enterprise Linux 8
Server	Same as indexer, all-in-one	-	-
Dashboard	Same as indexer, all-in-one	-	-
Agent	Installing Wazuh agents	-	Red Hat Enterprise Linux 8 x86_64, Windows 11 x86_64

Test description

0. Follow and read documentation links to test ISM policies in Wazuh Indexer:

https://documentation-dev.wazuh.com/v4.9.1-rc1/user-manual/wazuh-indexer/index-life-management.html https://opensearch.org/docs/latest/im-plugin/ism/index/

1. Create a retention policy using visual editor (10 m)

Create a retention policy using the visual editor of 10 minutes.

2. Create a retention policy using JSON editor (10 m)

Create a retention policy using the JSON editor of 10 minutes. With different index patterns from Step 1.

3. Applying the retention policy to alerts index

Apply the retention policies from the previous Steps.

4. Generate a new alert.

5. Validate that retention policies, checking the alert generated before does not exist.

Check the size of the files where the policies were applied, and check that after 10 minutes the size of the files decreases. And the alert generated in Step 4 does not exist.

6. Modify the time of both retention policies (Steps 1 and 2).

7. Apply and check that the modified policies work

Apply and check that the modified policies work, following once again Step 5 with the modified policies.

8. Roll Over

Known issues

There are no known issues.

Conclusions

Summarize the errors detected (Known Issues included). Illustrate using the table below. REMOVE CURRENT EXAMPLES:

Status	Test	Failure type	Notes
:black_circle:	Creating a retention policy using visual editor
:black_circle:	Creating a retention policy using json editor
:black_circle:	Applying the retention policy to alerts index
:black_circle:	Generate a new alert
:black_circle:	Verify that the retention policies worked
:black_circle:	Modify the time of both retention policies
:black_circle:	Apply and check that the modified policies work
:black_circle:	Roll Over

Feedback

We value your feedback. Please provide insights on your testing experience.

• Was the testing guideline clear? Were there any ambiguities?
  • Yes steps were clear and easy to follow. nonetheless lack of instructions regardin how to perform Roll over + Alias, what exactly is expected to be done and what should be the outcome of the test.
• Did you face any challenges not covered by the guideline?
  • **Yes, the guidelines do not contain verification steps to ensure the retention policy does work. Also no mention on how to perform rollback in the documentation. **
• Suggestions for improvement:
  • Create documentation on how to create and manage data streams and performing roll over. See Issue#5771
  • When creating a retention policy, set it to 10 minutes, and in the step of verifying whether the retention policy worked, check if the size of the files decreased after the 10 minutes.
  • Perform the step of verifying the retention policy after applying that policy and after creating the alert.
  • When modifying a retention policy, set it to 5 minutes, and in the step of verifying whether the retention policy worked, check if the size of the files decreased after the 5 minutes.

Reviewers validation

The criteria for completing this task is based on the validation of the conclusions and the test results by all reviewers.

All the checkboxes below must be marked in order to close this issue.

• [ ] @juliamagan
• [ ] @wazuh/devel-indexer

[joaquinsgi]

This issue goes on hold until it is finished: https://github.com/wazuh/wazuh/issues/26475.

[joaquinsgi]

This issue is back to in progress, because https://github.com/wazuh/wazuh/issues/26475 it is finished and more priority issues were finished.

[joaquinsgi]

I suggest the test should be like the following, see Test Description Step and Conclusions table:

End-to-End (E2E) Testing Guideline

• Documentation: Always consult the development documentation for the current stage tag at this link. Be careful because some of the description steps might refer to a current version in production, always navigate using the current development documention for the stage under test. Also, visit the following pre-release package guide to understand how to modify certain links and urls for the correct testing of the development packages.
• Test Requirements: Ensure your test comprehensively includes a full stack and agent/s deployment as per the Deployment requirements, detailing the machine OS, installed version, and revision.
• Deployment Options: While deployments can be local (using VMs, Vagrant, etc) or on the aws-dev account, opt for local deployments when feasible. For AWS access, coordinate with the DevOps team through this link.
• External Accounts: If tests require third-party accounts (e.g., GitHub, Azure, AWS, GCP), request the necessary access through the DevOps team here.
• Alerts: Every test should generate a minimum of one end-to-end alert, from the agent to the dashboard, irrespective of test type.
• Multi-node Testing: For multi-node wazuh-manager tests, ensure agents are connected to both workers and the master node.
• Package Verification: Use the pre-release package that matches the current TAG you're testing. Confirm its version and revision.
• Filebeat Errors: If you encounter errors with Filebeat during testing, refer to this Slack discussion for insights and resolutions.
• Known Issues: Familiarize yourself with previously reported issues in the Known Issues section. This helps in identifying already recognized errors during testing.
• Reporting New Issues: Any new errors discovered during testing that aren't listed under Known Issues should be reported. Assign the issue to the corresponding team (QA if unsure), add the Release testing objective and Urgent priority. Communicate these to the team and QA via the c-release Slack channel.
• Test Conduct: It's imperative to be thorough in your testing, offering enough detail for reviewers. Incomplete tests might necessitate a redo.
• Documentation Feedback: Encountering documentation gaps, unclear guidelines, or anything that disrupts the testing or UX? Open an issue, especially if it's not listed under Known Issues. Please answer the feedback section, this is a mandatory step.
• Format: If this is your first time doing this, refer to the format (but not necessarily the content, as it may vary) of previous E2E tests, here you have an example https://github.com/wazuh/wazuh/issues/13994.
• Status and completion: Change the issue status within your team project accordingly. Once you finish testing and write the conclusions, move it to Pending review and notify the @wazuh/devel-indexer team via Slack using the c-release channel. Beware that the reviewers might request additional information or task repetitions.
• For reviewers: Please move the issue to Pending final review and notify via Slack using the same thread if everything is ok, otherwise, perform an issue update with the requested changes and move it to On hold, increase the review_cycles in the team project by one and notify the issue assignee via Slack using the same thread.

For the conclusions and the issue testing and updates, use the following legend:

Status legend

• 🟢 All checks passed
• 🟡 Found a known issue
• 🔴 Found a new error

Issue delivery and completion

• Initial delivery: The issue's assignee must complete the testing and deliver the results by Sep 23, 2024 and notify the @wazuh/devel-indexer team via Slack using the c-release channel
• Review: The @wazuh/devel-indexer team will assign a reviewer and add it to the review_assignee field in the project. The reviewer must then review the test steps and results. Ensure that all iteration cycles are completed by Sep 24, 2024 date (issue must be in Pending final review status) and notify the QA team via Slack using the c-release channel.
• Auditor: The QA team must audit, validate the results, and close the issue by Sep 25, 2024.

Deployment requirements

Component	Installation	Type	OS
Indexer	Quickstart	-	Red Hat Enterprise Linux 8
Server	Same as indexer, all-in-one	-	-
Dashboard	Same as indexer, all-in-one	-	-
Agent	Installing Wazuh agents	-	Red Hat Enterprise Linux 8 x86_64, Windows 11 x86_64

Test description

1. Create a retention policy using visual editor (10 m)

Create a retention policy using the visual editor with 10 minutes Min Index Age. Follow the steps on: Creating a retention policy for Visual Editor

2. Create a retention policy using JSON editor (10 m)

Create a retention policy using the JSON editor with 10 minutes Min Index Age. Follow the steps on: Creating a retention policy for JSON Editor With different index patterns from Step 1, for example using wazuh-archives-*

3. Apply the retention policy to Alerts Index

4. Apply the retention policy to Archives Index

5. Generate a new alert and validate the retention policy from Step 1 worked

Checking the alert generated before does not exist after 10 minutes.

6. Validate that the retention policy from Step 2 worked

Checking the wazuh-archives where was applied the policy 10 minutes later confirms data has been deleted(reduced size and the number of documents).

7. Modify the Min Index Age of both retention policies (Steps 1 and 2).

8. Apply and check that the modified policies worked.

Following once again Step 5 with the modified policies.

Known issues

Conclusions

Summarize the errors detected (Known Issues included). Illustrate using the table below. REMOVE CURRENT EXAMPLES:

Status	Test	Failure type	Notes
:black_circle:	Create a retention policy using Visual editor
:black_circle:	Create a retention policy using JSON editor
:black_circle:	Apply the retention policy to Alerts Index
:black_circle:	Apply the retention policy to Archives Index
:black_circle:	Generate a new alert and validate the retention policy from Step 1 worked
:black_circle:	Validate that the retention policy from Step 2 worked
:black_circle:	Modify the Min Index Age of both retention policies
:black_circle:	Apply and check that the modified policies worked

Feedback

We value your feedback. Please provide insights on your testing experience.

• Was the testing guideline clear? Were there any ambiguities?

• Did you face any challenges not covered by the guideline?

Reviewers validation

The criteria for completing this task is based on the validation of the conclusions and the test results by all reviewers.

All the checkboxes below must be marked in order to close this issue.

• [ ] @juliamagan
• [ ] @wazuh/devel-indexer

[juliamagan]

Changes applied to E2E template.