Fix pytest errors test collection

[Rebits]

Related issue
#3969

Description

This PR fixes the framework and testing errors detected at collecting time. In addition, it adds a new parameter to toggle the deselection of tests according to the environment where pytest is launched.

Added

• Include new parameter to toggle environment tests deselection

Updated

• Fixed errors during test collection

Testing performed

• [x] --avoid-platform-based-deselection: https://github.com/wazuh/wazuh-qa/pull/3972#issuecomment-1437411031
• [x] Tests collection: https://github.com/wazuh/wazuh-qa/pull/3972#issuecomment-1437407921
• [x] Get service changes
• [x] Test agentd
• [x] Test analysisd
• [x] Test GCloud
• [x] Test Github
• [x] Test integratord

[Rebits]

Tests collection

Integration

Output: pytest-it-collection

System

Output: pytest-system-collection

Performance

Output: pytest-performance-collection

Reliability

Output: pytest-reliability-collection

E2E

Output: pytest-e2e-collection

[Rebits]

Avoid platform-based deselection parameters

All IT collection ignoring environment limitations

Output: output_collection

Note Notice that there is no deselected tests

[Rebits]

22/05/2023

Build: https://ci.wazuh.info/job/Test_integration_launcher/139/

[Rebits]

23/05/2023

The following tests fail after changes:

• test_gcloud/test_functionality/test_rules.py::test_rules[get_configuration0] -> This failure is not related with the changes: https://ci.wazuh.info/job/Test_integration/39161/console (Test pass in the second iteration)
• test_agentd/test_agentd_parametrized_reconnections

---

It seems that test_agentd_parametrized_reconnections has been affected by the changes, further research is required

[Rebits]

26/06/2023

• Changes base branch to 4.4.5
• Resolved conflicts
• Fix minor errors in library changes
• Fix pep8 in changed files

Currently testing changes

[Rebits]

27/06/2023

• Fix error in integratord module
• Testing all the suite: https://ci.wazuh.info/job/Test_integration_launcher/157/

[Rebits]

28/06/2023

• https://ci.wazuh.info/job/Test_integration/40629/
• All suite: https://ci.wazuh.info/job/Test_integration_launcher/158/
  • :red_circle: Syscollector: Expected https://github.com/wazuh/wazuh-qa/issues/4230
  • :red_circle: Vulnerability Detector: Expected https://github.com/wazuh/wazuh-qa/issues/4231

[mauromalara]

@Rebits

In both tests, here and here I can see an error before the collection (it seems to appear only in the integration tests when there are deselected tests), do you know what is this about?

platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-0.13.1
rootdir: /home/rebits/Wazuh/wazuh-qa/3969-fix-pytest-collection/tests/integration, configfile: pytest.ini
plugins: html-3.1.1, testinfra-5.0.0, json-report-1.5.0, metadata-2.0.1, docgen-1.3.0
Error: 'NoneType' object is not subscriptable <------- HERE! 🐛
Using default values
collected 61906 items

---

Also, there is another error in the Performance tests collection, please check it:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/_pytest/main.py", line 268, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/_pytest/main.py", line 321, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/_pytest/main.py", line 332, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/_pytest/main.py", line 660, in perform_collect
INTERNALERROR>     hook.pytest_collection_modifyitems(
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/rebits/.local/lib/python3.10/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/rebits/Wazuh/wazuh-qa/tests/performance/test_api/conftest.py", line 176, in pytest_collection_modifyitems
INTERNALERROR>     item.user_properties.extend([(key, value) for key, value in item.callspec.params['test_case'].items()])

[Rebits]

In both tests, https://github.com/wazuh/wazuh-qa/pull/3972#issuecomment-1437411031 and https://github.com/wazuh/wazuh-qa/pull/3972#issuecomment-1437407921 I can see an error before the collection (it seems to appear only in the integration tests when there are deselected tests), do you know what is this about?

That was a previous state of the development and not the final result. That output demonstrates the functionality of the new parameter --avoid-platform-based-deselection, which enables the collection of all tests, regardless of platform deselection

Regarding the indexation of evidences for correct collection, it is not strictly required due to the implementation of the new Tests collection action. However, I index them in this comment if needed:

• List of QA tests

[Rebits]

Regarding these comments:

Why are you modifying a test? Isn't it out of scope?

I acknowledge that I could have done a better job in clearly exposing the problem. The primary objective of this development is to enable error-free test collection. However, we are facing several challenges:

1. The current framework is tightly coupled with the specific host it is used on, depending on the OS and assuming the presence of a Wazuh component installed. This will become increasingly problematic in the future, especially for multi-host testing (E2E, System, etc.) that may not necessarily be launched in a Wazuh environment. To address this issue, we are working on changes such as modifying the get_version and get_service functions.
2. Many tests do not adhere to best practices, such as breaking code outside of fixtures and the tests themselves. As a result, multiple errors occur during test collection.

To illustrate the current status of the IT collection in the repository, I index the error output:

==================================== ERRORS ====================================
____ ERROR collecting test_agentd/test_agentd_parametrized_reconnections.py ____
tests/integration/test_agentd/test_agentd_parametrized_reconnections.py:132: in <module>
    set_debug_mode()
tests/integration/test_agentd/test_agentd_parametrized_reconnections.py:123: in set_debug_mode
    with open(local_int_conf_path) as local_file_read:
E   PermissionError: [Errno 13] Permission denied: '/var/ossec/etc/local_internal_options.conf'
___________ ERROR collecting test_agentd/test_agentd_reconnection.py ___________
tests/integration/test_agentd/test_agentd_reconnection.py:133: in <module>
    set_debug_mode()
tests/integration/test_agentd/test_agentd_reconnection.py:124: in set_debug_mode
    with open(local_int_conf_path, 'r') as local_file_read:
E   PermissionError: [Errno 13] Permission denied: '/var/ossec/etc/local_internal_options.conf'
_ ERROR collecting test_fim/test_registry/test_registry_basic_usage/test_basic_usage_registry_duplicated_entries.py _
tests/integration/test_fim/test_registry/test_registry_basic_usage/test_basic_usage_registry_duplicated_entries.py:65: in <module>
    @pytest.mark.skipif(get_version() != 'v4.2.3', reason="This test fails by wazuh/wazuh#6797, It was fixed on v4.2.3")
../../../.local/lib/python3.10/site-packages/wazuh_testing-4.5.0-py3.10.egg/wazuh_testing/tools/utils.py:134: in get_version
    f = open('../../version.json')
E   FileNotFoundError: [Errno 2] No such file or directory: '../../version.json'
_______ ERROR collecting test_gcloud/test_functionality/test_interval.py _______
tests/integration/test_gcloud/test_functionality/test_interval.py:98: in <module>
    truncate_file(LOG_FILE_PATH)
../../../.local/lib/python3.10/site-packages/wazuh_testing-4.5.0-py3.10.egg/wazuh_testing/tools/file.py:79: in truncate_file
    with open(file_path, 'w'):
E   PermissionError: [Errno 13] Permission denied: '/var/ossec/logs/ossec.log'
_______ ERROR collecting test_gcloud/test_functionality/test_logging.py ________
import file mismatch:
imported module 'test_logging' has this __file__ attribute:
  /home/rebits/Wazuh/wazuh-qa/4.5.0/tests/integration/test_analysisd/test_limit_eps/test_logging.py
which is not the same as the test file we want to collect:
  /home/rebits/Wazuh/wazuh-qa/4.5.0/tests/integration/test_gcloud/test_functionality/test_logging.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
_____ ERROR collecting test_gcloud/test_functionality/test_max_messages.py _____
tests/integration/test_gcloud/test_functionality/test_max_messages.py:105: in <module>
    truncate_file(LOG_FILE_PATH)
../../../.local/lib/python3.10/site-packages/wazuh_testing-4.5.0-py3.10.egg/wazuh_testing/tools/file.py:79: in truncate_file
    with open(file_path, 'w'):
E   PermissionError: [Errno 13] Permission denied: '/var/ossec/logs/ossec.log'
________ ERROR collecting test_gcloud/test_functionality/test_rules.py _________
tests/integration/test_gcloud/test_functionality/test_rules.py:96: in <module>
    truncate_file(LOG_FILE_PATH)
../../../.local/lib/python3.10/site-packages/wazuh_testing-4.5.0-py3.10.egg/wazuh_testing/tools/file.py:79: in truncate_file
    with open(file_path, 'w'):
E   PermissionError: [Errno 13] Permission denied: '/var/ossec/logs/ossec.log'
_______ ERROR collecting test_github/test_configuration/test_invalid.py ________
import file mismatch:
imported module 'test_invalid' has this __file__ attribute:
  /home/rebits/Wazuh/wazuh-qa/4.5.0/tests/integration/test_gcloud/test_configuration/test_invalid.py
which is not the same as the test file we want to collect:
  /home/rebits/Wazuh/wazuh-qa/4.5.0/tests/integration/test_github/test_configuration/test_invalid.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
___________ ERROR collecting test_integratord/test_alerts_reading.py ___________
tests/integration/test_integratord/test_alerts_reading.py:93: in <module>
    t1_config_params = replace_webhook_url(t1_cases_ids, t1_config_params)
tests/integration/test_integratord/test_alerts_reading.py:69: in replace_webhook_url
    configurations[i]['WEBHOOK_URL'] = global_parameters.slack_webhook_url
E   AttributeError: 'Parameters' object has no attribute 'slack_webhook_url'
______ ERROR collecting test_office365/test_configuration/test_invalid.py ______
import file mismatch:
imported module 'test_invalid' has this __file__ attribute:
  /home/rebits/Wazuh/wazuh-qa/4.5.0/tests/integration/test_gcloud/test_configuration/test_invalid.py
which is not the same as the test file we want to collect:
  /home/rebits/Wazuh/wazuh-qa/4.5.0/tests/integration/test_office365/test_configuration/test_invalid.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
_________________ ERROR collecting test_wpk/test_wpk_agent.py __________________
tests/integration/test_wpk/test_wpk_agent.py:97: in <module>
    raise Exception("The WPK package version must be defined by parameter. See README.md")
E   Exception: The WPK package version must be defined by parameter. See README.md
________________ ERROR collecting test_wpk/test_wpk_manager.py _________________
tests/integration/test_wpk/test_wpk_manager.py:75: in <module>
    WPK_REPOSITORY_4x = global_parameters.wpk_package_path[0]
E   TypeError: 'NoneType' object is not subscriptable
__________ ERROR collecting test_wpk/test_wpk_manager_task_states.py ___________
tests/integration/test_wpk/test_wpk_manager_task_states.py:70: in <module>
    WPK_REPOSITORY_4x = global_parameters.wpk_package_path[0]
E   TypeError: 'NoneType' object is not subscriptable
=============================== warnings summary ===============================
../../../.local/lib/python3.10/site-packages/google/pubsub_v1/services/publisher/client.py:19
  /home/rebits/.local/lib/python3.10/site-packages/google/pubsub_v1/services/publisher/client.py:19: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
    from distutils import util

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
ERROR tests/integration/test_agentd/test_agentd_parametrized_reconnections.py
ERROR tests/integration/test_agentd/test_agentd_reconnection.py - PermissionE...
ERROR tests/integration/test_fim/test_registry/test_registry_basic_usage/test_basic_usage_registry_duplicated_entries.py
ERROR tests/integration/test_gcloud/test_functionality/test_interval.py - Per...
ERROR tests/integration/test_gcloud/test_functionality/test_logging.py
ERROR tests/integration/test_gcloud/test_functionality/test_max_messages.py
ERROR tests/integration/test_gcloud/test_functionality/test_rules.py - Permis...
ERROR tests/integration/test_github/test_configuration/test_invalid.py
ERROR tests/integration/test_integratord/test_alerts_reading.py - AttributeEr...
ERROR tests/integration/test_office365/test_configuration/test_invalid.py
ERROR tests/integration/test_wpk/test_wpk_agent.py - Exception: The WPK packa...
ERROR tests/integration/test_wpk/test_wpk_manager.py - TypeError: 'NoneType' ...
ERROR tests/integration/test_wpk/test_wpk_manager_task_states.py - TypeError:...
!!!!!!!!!!!!!!!!!!! Interrupted: 13 errors during collection !!!!!!!!!!!!!!!!!!!
= 61181/61831 tests collected (650 deselected), 13 errors in 107.30s (0:01:47) =

---

Regarding if some changes belong to the development, let's consider the test test_agentd_reconnection. In this test, there is a call to the function set_debug_mode which is not encapsulated within either a fixture or the test itself. As a result, this leads to the following error:

___________ ERROR collecting test_agentd/test_agentd_reconnection.py ___________
tests/integration/test_agentd/test_agentd_reconnection.py:133: in <module>
    set_debug_mode()
tests/integration/test_agentd/test_agentd_reconnection.py:124: in set_debug_mode
    with open(local_int_conf_path, 'r') as local_file_read:
E   PermissionError: [Errno 13] Permission denied: '/var/ossec/etc/local_internal_options.conf'
_ ERROR collecting test_fim/test_registry/test_registry_basic_usage/test_basic_usage_registry_duplicated_entries.py _
tests/integration/test_fim/test_registry/test_registry_basic_usage/test_basic_usage_registry_duplicated_entries.py:65: in <module>
    @pytest.mark.skipif(get_version() != 'v4.2.3', reason="This test fails by wazuh/wazuh#6797, It was fixed on v4.2.3")
../../../.local/lib/python3.10/site-packages/wazuh_testing-4.5.0-py3.10.egg/wazuh_testing/tools/utils.py:134: in get_version

So I am afraid that in most cases it would be necessary to change some tests.