certification-tool icon indicating copy to clipboard operation
certification-tool copied to clipboard

[Bug] SDK Python tests fails when running using cli.sh

Open Kshitijjain21 opened this issue 8 months ago • 1 comments

Describe the bug

When running the SDK python tests using cli.sh the test raises a Value error ValueError: Level 'PYTHON_TEST' does not exist . This is observed using test harness version v2.10+spring2024.

Steps to reproduce the behavior

The error can be observed be running the following command on the test harness (version v2.10+spring2024)

  1. Make the DUT in commissioning mode Eg. Run all-cluster-app, lighting app or any other application.
  2. ./cli.sh run-tests --project-id 1 -s '{"SDK Python Tests":{"Python Testing Suite":{"TC_ACE_1_3":1}}}'

Expected behavior

Creating new test run with title: 2024-06-13-13:13:28
Starting Test run: Title: 2024-06-13-13:13:28, id: 18
Test Run [EXECUTING]
  - Python Testing Suite [EXECUTING]
=======================================
Make sure the DUT is in Commissioning Mode
  1: DONE
  2: FAILED
Please enter a number for an option above: 
1
=======================================
      - TC-ACE-1.3 [EXECUTING]
            - Start Python test [EXECUTING]
Traceback (most recent call last):
  File "/home/ubuntu/certification-tool/cli/app/main.py", line 47, in <module>
    root()
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/certification-tool-cli-Bd4yKGBQ-py3.10/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/certification-tool-cli-Bd4yKGBQ-py3.10/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/certification-tool-cli-Bd4yKGBQ-py3.10/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/certification-tool-cli-Bd4yKGBQ-py3.10/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/certification-tool-cli-Bd4yKGBQ-py3.10/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/ubuntu/certification-tool/cli/app/async_cmd.py", line 28, in wrapper
    return asyncio.run(f(*args, **kwargs))
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/ubuntu/certification-tool/cli/app/commands/run_tests.py", line 65, in run_tests
    await socket_task
  File "/home/ubuntu/certification-tool/cli/app/test_run/websocket.py", line 71, in connect_websocket
    await self.__handle_incoming_socket_message(socket=socket, message=message_obj)
  File "/home/ubuntu/certification-tool/cli/app/test_run/websocket.py", line 82, in __handle_incoming_socket_message
    self.__handle_log_record(message.payload)
  File "/home/ubuntu/certification-tool/cli/app/test_run/websocket.py", line 128, in __handle_log_record
    logger.log(record.level, record.message)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/certification-tool-cli-Bd4yKGBQ-py3.10/lib/python3.10/site-packages/loguru/_logger.py", line 1997, in log
    __self._log(level_id, static_level_no, False, __self._options, __message, args, kwargs)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/certification-tool-cli-Bd4yKGBQ-py3.10/lib/python3.10/site-packages/loguru/_logger.py", line 1886, in _log
    raise ValueError("Level '%s' does not exist" % level_id) from None
ValueError: Level 'PYTHON_TEST' does not exist

Log files

No response

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

The fix is to add the following lines in cli/app/test_run/logging.py

# Add custom logger for python tests
PYTHON_TEST_LEVEL = "PYTHON_TEST"
logger.level(PYTHON_TEST_LEVEL, no=22, icon="🐍", color="<cyan>")

Happy to create a PR for this. Just want to confirm that are the steps to contribute to this repo same as steps mention in the Matter SDK ?

Kshitijjain21 avatar Jun 13 '24 14:06 Kshitijjain21