Add E2E and unit tests to the front-end side and improve back-end tests

[jesusgn90]

Front-end side

Our application is growing day by day and the manual testing is going to be unmaintainable. For this reason, I think we should add unit tests for the frontend side (all the Angular.js and React.js part) and we should also add E2E tests.

What's an E2E test?

Simulating human actions such as "Open Overview Welcome, then apply a filter, the visualizations must change depending on the filter"

What's a unit test

Automatic tests for all the logic under the views, this means we can properly test a module or a directive at the function level.

For example the function addLevel(parameters) is expected to receive no more than one parameter, if it's an integer it should return the integer x 2, then, if it's not an integer it should return false.

The example function may have multiple unit tests such as: "call the function with zero parameters", "call the function with a valid parameter and check the output", "call the function with an invalid parameter and check the output", etc..

First iteration

• [ ] Decide which framework we should use for E2E testing on the front-end side.
• [ ] Decide which framework we should use for unit testing on the front-end side.
• [ ] Create a fake but functional E2E test.
• [ ] Create a fake but functional unit test.

Second iteration

• [ ] Cover most common visualize actions with E2E tests.
• [ ] Cover most common visualize actions with unit tests.

Third iteration

• [ ] Cover the rest of the frontend parts with E2E tests.
• [ ] Cover the rest of the frontend parts with unit tests.

Fourth iteration

• [ ] Add coverage reports using external libraries for all the frontend side.

Example frameworks for front-end testing:

• https://www.cypress.io/
• https://www.protractortest.org/#/

Server side

For the server, we've already made some tests using Mocha, see https://github.com/wazuh/wazuh-kibana-app/tree/master/test/server for details. However, they are probably outdated and some of them are broken. For this reason, we must improve and create new tests for the server side too.

First iteration

• [ ] Review and fix all existing tests for the server.
• [ ] Cover all the server side with more Mocha tests.

Second iteration

• [ ] Add coverage reports using external libraries for all the server side.

[jesusgn90]

From @jsanchez91:

E2E Testing

I am looking in the official documentation of Angular about of E2E testing, they recommend use Protractor.js. Protractor is a program for Node.js, it uses the Selenium's WebDriver to simulate user actions in the web browsers. For write the tests, Protactor uses Jasmine syntax, it is a clean, obvious syntax so that you can easily write tests.

EXAMPLE:

describe('sum function tests' () => {
    if('one plus two is three', () => {
        expect(sum(1 + 2)).toBe(3);
    });
    if('one plus two not is four', () => {
        expect(sum(1 + 2)).not.toBe(4);
    });
    if('two plus two is four', () => {
        expect(sum(2 + 2)).toBe(4);
    });    
});

Protractor needs the Java Development Kit (JDK) installed to run the Selenium Server.

For running in a server side, is necessary to install a headless browser, like Google Chrome or Mozilla Firefox. I tried to do an example for verifying all requirements to execute the test, these are the inputs what I used in a Centos server.

# Install Protractor
cd ./kibana/plugins/wazuh
yarn add protractor

# Install JDK
sudo yum install java-1.8.0-openjdk

# Install Google Chrome
sudo vi /etc/yum.repos.d/google-chrome.repo

# Add this in the editor and save
[google-chrome]
name=google-chrome
baseurl=http://dl.google.com/linux/chrome/rpm/stable/x86_64
enabled=1
gpgcheck=1
gpgkey=https://dl.google.com/linux/linux_signing_key.pub

# Execute yum to install
sudo yum install google-chrome-stable

# Update and execute the Selenium WebDriver
yarn webdriver-manager update
yarn webdriver-manager start

I executed the example of the official documentation of Protractor to verify the environment.

Protractor needs two files to run, a spec file and a configuration file, the content of this files is the next:

spec.js

describe('Protractor Demo App', function() {
    it('should have a title', function() {
        browser.get('http://juliemr.github.io/protractor-demo/');

        expect(browser.getTitle()).toEqual('Super Calculator');
    });
});

conf.js

exports.config = {
    framework: 'jasmine',
    seleniumAddress: 'http://localhost:4444/wd/hub',
    specs: ['spec.js'],
    capabilities: {
        browserName: 'chrome',
        chromeOptions: {
            args: ["--headless", "--disable-gpu", "--window-size=800,600"]
        }
    }
}

Finally to execute the test I used this command:

yarn protractor conf.js

# Output
[10:22:10] I/launcher - Running 1 instances of WebDriver
[10:22:10] I/hosted - Using the selenium server at http://localhost:4444/wd/hub
Started
.


1 spec, 0 failures
Finished in 0.728 seconds

[10:22:12] I/launcher - 0 instance(s) of WebDriver still running
[10:22:12] I/launcher - chrome #01 passed
Done in 1.82s.

[jsanchez91]

For task Create a fake but functional E2E test of first iteration I wrote a few test.

These check the elements in main menu of Wazuh application, verifying the title names and the correct controller selection when do click in its.

The code has been upload in the commit: df450f2

To run the E2E tests you will need install all dependencies mentioned in the previous comment of this issue (Protractor, JDK and Google Chrome) and execute this command from Wazuh app path.

cd ./kibana/plugins/wazuh
yarn protractor test/e2e/conf.js

In the next iterations we will add more tests, refactor the code and modify the structure of the folder.

[jsanchez91]

Currently, the Wazuh app for Kibana has some tests made with the Mocha framework. Kibana has support to make the tests using Mocha and Jest.

In the official documentation of Kibana they recommend using Jest. This framework has a lot of functions to do mock and simulate inputs.

The current tests made with Mocha was been broken, only 21 passing and 18 falling.

Because the E2E tests are more important for us, we leave the unit test for later.

[jsanchez91]

I take these issues #1384, #1541 for creating this structure in the test/e2e folder:

This structure corresponds to the main menu of Wazuh app

In the future, it is possible that we change the structure to add more specified tests like which directive, controller or components tests.

[jsanchez91]

I made a push in the branch issue-1539 to add one commit.

In this commit, I include the structure of the folder test/e2e and new tests inside the test/e2e/overview/spec.js.

These tests check if all panels in the view Overview / Security events load correctly.

When I wrote the tests, I had a problem with the first test in the file. This test failed because of the health_check controller didn´t finish his execution. For solving this problem, I add the first test to check the current URL contain the word overview.

it('wait to finish the healt check', async () => {
    let loading = true;
    while(loading){
        await browser.getCurrentUrl().then((text)=>{
            if (text.includes('overview')){
                loading = false;
                expect(browser.getCurrentUrl()).toContain('overview');
            }
        })
    }
});

[jsanchez91]

I had a problem with some tests cases, for example:

Test case: Check the initial number of checks of the health check when opening the app. It should be 4 + 1 extra for the known fields.

I am looking for a method to make this test because I can not stop the flow of the application.

This is the description of the test problem:

1. browser.get('/app/wazuh')
2. The webdriver sends a signal to the web browser to make the request.
3. The web browser starts to build the page.
4. Get the page elements on the test.

// Get all rows of the table
const rows = element(by.tagName('table')).all(by.tagName('tr'))

The web browser continues building the web page, its doesn´t stop.

5. I created 5 expect condition, one per row:

expect(table.get('0').element(by.css('td > i.fa-check')).isPresent()).toBe(true);
expect(table.get('1').element(by.css('td > i.fa-check')).isPresent()).toBe(true);
expect(table.get('2').element(by.css('td > i.fa-check')).isPresent()).toBe(true);
expect(table.get('3').element(by.css('td > i.fa-check')).isPresent()).toBe(true);
expect(table.get('4').element(by.css('td > i.fa-check')).isPresent()).toBe(true);

The problem here is when I use the method .isPresent() the test makes an async request to the webdriver, and waits for the response, this repeats 5 times. In the last petitions, the web browser finishes loading the page and change the path for /app/wazuh#/overview, when the test try to get the elements in the row, they aren´t been and the test fail

To see the output with more detail, I use this code:

table.get('0').element(by.css('td > i.fa-check')).isPresent().then((status) => {
    console.log(`Row 1: ${status}`);
});
table.get('1').element(by.css('td > i.fa-check')).isPresent().then((status) => {
    console.log(`Row 2: ${status}`);
});
table.get('2').element(by.css('td > i.fa-check')).isPresent().then((status) => {
    console.log(`Row 3: ${status}`);
});
table.get('3').element(by.css('td > i.fa-check')).isPresent().then((status) => {
    console.log(`Row 4: ${status}`);
});
table.get('4').element(by.css('td > i.fa-check')).isPresent().then((status) => {
    console.log(`Row 5: ${status}`);
});

// Output:
Row 1: true
Row 2: true
Row 3: true
Row 4: false
Row 5: false

I am looking for a solution to guarantee the execution of the tests. The best I found is separate the tests and restart the web browser after finish the expected condition. The problem is the last test fails because takes more time to change green. This is one separate test example:

it('Check Wazuh API connection is green', async () =>{
  await browser.get(`http://localhost:5601/app/wazuh#/`);
  browser.waitForAngular();
  const table = element(by.tagName('tbody')).all(by.tagName('tr'));
  expect(table.get('0').element(by.css('td:nth-child(1)')).getText()).toEqual('Check Wazuh API connection');
  expect(table.get('0').element(by.css('td > i.fa-check')).isPresent()).toBe(true);
  browser.restart();
});

[jsanchez91]

We discovered the method of Kibana to tests plugins and we remade all environment.

Kibana calls functional tests to the e2e tests, in its documentation you can see more info about the functional testing in Kibana and plugins.

• Kibana functional testing
• Functional Tests for Plugins
• Example plugin with functional tests

---

I reset the branch issue-1539 and added new commits.

These are the steps what I did to run the tests:

In first time, my environment to make the tests are a Centos server with Kibana 7.2.0. To run the tests only need installed Google Chrome browser, in some comments up I explain how to install in Centos.

In case you connect to the machine with ssh, you can use the browser in headless mode executing this command in the server terminal:

export TEST_BROWSER_HEADLESS=0

But if you prefer, you can try to use x11 forwarding:

• Config x11 forwarding
• x11 forwarding from vagrant

To execute the tests is necessaries start a special instance of Kibana for testing. From the Wazuh plugin path ./kibana/plugins/wazuh using the branch issue-1539 execute this command:

yarn test:ui:server

This instance starts an elastic search service in the port 9220, it is necessary to change in FileBeat this to see the alerts in the tests.

sudo vi /etc/filebeat/filebeat.yml

# Find the property `output.elasticsearch:` and change the `hosts:` value
output.elasticsearch:
  hosts: ['http://localhost:9220'] # Change in this line
  #pipeline: geoip
  indices:
    - index: 'wazuh-alerts-3.x-%{+yyyy.MM.dd}'

# Save the file and restart the FileBeat service
sudo systemctl restart filebeat

# Use this command to check that everything is fine.
sudo filebeat test output

If you get this error: ERROR Connection marked as failed because the onConnect callback failed: This Beat requires the default distribution of Elasticsearch. Please upgrade to the default distribution of Elasticsearch from elastic.co, or downgrade to the oss-only distribution of beats Change your FileBeat for an oss version

It's the moment to execute the tests, from the Wazuh path again ./kibana/plugins/wazuh execute this:

yarn test:ui:runner

If all is fine you get this output:

[jsanchez91]

I add more commits to the branch issue-1539.

The code of these commits introduces two new concepts: fixtures and test_subjects.

Fixtures

When made tests the data input is very important, if the data change, maybe the test will break. For preventing this, exists the fixture files. In Elastic stack you can export an index and loaded in the test execution, to have always the same data input.

Kibana has the function script/es_archive to this purpose, for export an index to the Wazuh app use this command in the terminal:

cd ./kibana/plugins/wazuh

node ../../scripts/es_archive save wazuh_alerts wazuh-alerts-3.x-2019.07.02

• The first parameter save is the command, other commands are load or unload.
• The second is the name to save index.
• And the last is the index name in Elastic.

In the code to load the fixture in the tests use the service esArchiver, for example:

export default function ({ getService, loadTestFile }) {
  const esArchiver = getService('esArchiver'); // Import the service

  describe('Load fixture ', function () {

    before(async function () {
      await esArchiver.load('wazuh_alerts'); // Load the fixture
    });
    
    loadTestFile(require.resolve('./_test_app'));
  });
}

For more info consult this kibana issue https://github.com/elastic/kibana/pull/10359

test_subjects

Kibana has an HTML property to find elements in the test, this is data-test-subj.

To locale elements in the DOM you can find the element using its CSS Selector or Tag name, with the find service. Kibana recommends using the test_subject service.

• test_subject service
• find service

[jsanchez91]

I start the made the tests for these requisites:

Index patterns

• [x] Should have wazuh-alerts-3.x-* and wazuh-monitoring-3.x-*
• [x] None of the index-patterns should be the default

API management checks

• [x] Click on the Wazuh button on the left bar on the Kibana interface

  • [x] Should take you to Settings and warn you there are no API credentials available
  • [x] If it's a clean install, should appear "No API" on the top right corner
  • [x] Should appear the index pattern on the top right corner
  • [x] The extensions tab should be disabled

• [x] Filling "Add API" form badly once per every form field. Should appear the appropriate error message for every wrong inserted form field

• [x] Filling "Add API" form correctly Should connect successfully and show the data in the upper right corner

• [x] Check manager button right after inserting API credentials. Should success and not modify anything on the fields

• [ ] :negative_squared_cross_mark: Check the currently active extensions. Should be the same as the config.yml file

I move this test to the overview's tests

• [x] Insert a new API and check everyone with the Check button Should not change the currently selected API

• [x] Edit an API entry. The API is edited properly

• [x] Press F5 to reload the page. Should reload properly the currently active tab (Settings)

• [x] Go to a new tab (Management). After the health check should open the selected

• [x] Delete all the existing APIs

  • [x] The extensions tab should be disabled
  • [x] The top menu bar should be updated indicating that there's no API selected

[jsanchez91]

I will continue with these requirements.

Basic functions checks (Overview and Agents)

• [ ] Check the initial number of checks of the health check when opening the app. Should be 4 + 1 extra for the known fields.

Maybe this test is not possible to do, I leave it for later.

• [x] Click in Overview/Agents tab and select a proper time range
  • [ ] Should appear a loading bar and disappear after finishing loading

Maybe this test is not possible to do, I leave it for later.

• [x] Data should appear if there are alerts
• [x] Visualizations should appear correctly without errors
• [x] Click Overview/Agent -> Discover subtab. Should appear selected the wazuh-alerts-3.x index pattern only
• [x] Click on a rule ID on the Discover tab. It should open the Ruleset detail tab for that rule ID
• [x] Go to Dashboard subtab and activate a filter. The filter should be working
• [x] Go again to Discover subtab. The filter should be still working and appear as a chip
• [x] Go back to Dashboard and remove the filter. The visualizations should reload properly after deleting the filter
• [x] Press F5 to reload the page. The filters shouldn't keep applied unless they're pinned
• [x] Click several times the app buttons and tabs while you're on the same tab
  • [x] Filters should persist and not disappear
  • [x] The visualizations watchers should not lose data
• [x] Search something on the search bar, press Enter, remove the content and press Enter again. Should always perform correctly the search and change the visualizations accordingly.
• [x] Go to Settings and select different extensions, and then go back to Overview/Agents. The extensions configuration should keep on Overview.
• [x] Go to Settings, select a different API with different extensions and go back to Overview/Agents. Should keep its own configuration between API selections
• [x] Disable all the extensions, go to General and go back to Settings. The extensions should not be restored on any of the inserted APIs
• [x] Type something and press Enter on the different Discover search bars. After pressing Enter, the search should start
• [x] Go to Manager/Reporting prior to generating any report. Should appear a card about no reports available yet
• [ ] Click on the Report button on different tabs on Overview/Agents. After a wait, a new report should appear on Manager/Reporting
• [ ] Download a report on Management/Reporting
  • [ ] The report should be downloaded successfully
  • [ ] The report has to be related to the tab where it was generated
  • [ ] The report visualizations shouldn't have scrollbars
• [ ] Enable the Auto-Refresh functionality on Overview/Agents (Dev tools opened). The functionality should work properly
• [ ] Press F5 while having Auto-Refresh enabled Should keep working properly after refreshing the page.
• [ ] With the Auto-Refresh functionality enabled, go to Management (Dev tools opened). The functionality should stop working.
• [ ] From the Welcome screen, go to General, go back to Welcome and open again the General tab. The tab should reload properly after coming from Welcome
• [ ] Go from General to Welcome, then VirusTotal, then to Welcome, and then go again to General. Visualizations should load properly if there's actual data to show
• [ ] Enable the cluster on the Manager and open the app again. The app should load properly without errors
• [ ] The Overview/Agents tabs should work properly with the "cluster.name" filter enabled
• [ ] Check the "Surrounding documents" buttons on Overview/Agents/Discover tabs. Both buttons should properly work
• [ ] Add a simple filter and go to Discover, then back to Dashboard. The new filter should not be moved to the first position on the filter list

[jsanchez91]

I create a new entry in the wiki of the repository:

https://github.com/wazuh/wazuh-kibana-app/wiki/How-to-create-and-run-e2e-tests

I have pending push a new commit with the reorganized structure of the tests.

[jsanchez91]

I push some fixed and reorganize the code in the branch issue-1539

[jsanchez91]

Added 17 new tests in the file apps/overview/_welcome.js to cover these tasks:

• [x] Go to Settings and select different extensions, and then go back to Overview/Agents. The extensions configuration should keep on Overview.
• [x] Go to Settings, select a different API with different extensions and go back to Overview/Agents. Should keep its own configuration between API selections

In additional commits, move some tests to new files.

[jsanchez91]

Type something and press Enter on the different Discover search bars. After pressing Enter, the search should start

This task is divided into some tests:

• [x] Security Events
• [ ] Integrity monitoring
• [ ] Policy monitoring
• [ ] System auditing
• [ ] System SCA
• [ ] Vulnerabilities
• [ ] PCI DSS
• [ ] GDPR
• [ ] agents

[jsanchez91]

I add code documentation to the page_object and the services files

[jsanchez91]

I change the environment where the tests are run. I deleted test: ui: server from the package.json file. To run the test, it is necessary to add two environment variables to run the tests against an instance of Elasticsearch and Kibana.

export TEST_KIBANA_URL=https://kibana:[email protected]:443

export TEST_ES_URL=https://elastic:[email protected]:9200