thin-edge.io icon indicating copy to clipboard operation
thin-edge.io copied to clipboard

Published 20 hours ago • verified publisher icon thin-edge

Create Robotframework Automated Tests transfered from PySys

Open gligorisaev opened this issue 2 years ago • 6 comments

Proposed changes

tedge Testcases that are transfered

  • call_tedge_-V - Transferred to call_tedge.robot
  • call_tedge_-h - Transferred to call_tedge.robot
  • call_tedge_-h_-V - Transferred to call_tedge.robot
  • call_tedge_config_help - Transferred to call_tedge.robot
  • call_tedge_config_list - Will be covered in other E2E Test
  • tedge_agent_user_sudo_access - Obsolete
  • tedge_software_update - Will be covered in other E2E Test

tedge_connect Testcases that are transfered

  • c8y_restart_bridge - Removed
  • tedge_connect_test_negative - Transferred to tedge_connect_test.robot
  • tedge_connect_test_positive - Transferred to tedge_connect_test.robot
  • tedge_connect_test_sm_services - Transferred to tedge_connect_test.robot
  • tedge_disconnect_test_sm_services - Transferred to tedge_connect_test.robot

Types of changes

  • [ ] Bugfix (non-breaking change which fixes an issue)
  • [ ] New feature (non-breaking change which adds functionality)
  • [x] Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
  • [ ] Documentation Update (if none of the other choices apply)
  • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

#1255 #1256

Checklist

  • [x] I have read the CONTRIBUTING doc
  • [x] I have signed the CLA (in all commits with git commit -s)
  • [ ] I ran cargo fmt as mentioned in CODING_GUIDELINES
  • [ ] I used cargo clippy as mentioned in CODING_GUIDELINES
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • [ ] I have added necessary documentation (if appropriate)

Further comments

gligorisaev avatar Aug 04 '22 08:08 gligorisaev

So I don't know much about that transfer from pytest to robot, but from what I see this is not yet integrated in the github actions, so that's one point that is missing...

matthiasbeyer avatar Aug 04 '22 13:08 matthiasbeyer

So I don't know much about that transfer from pytest to robot, but from what I see this is not yet integrated in the github actions, so that's one point that is missing...

It's not yet decided how and where we want to run the RobotFramework tests. The current pain points I know of using PySys and GitHub Actions are

  • GitHub hosted runner cuts internet connection from time to time.
  • GitHub self-hosted runner - security. We did have many failure test results because the environment was not correct. Of course, we don't want to continue like that.

How? After merge? Nightly? I'd prefer to have it nightly workflow and run these tests from there, but haven't reached to consensus yet.

Where? GitHub hosted runner are really right place? Also, we can create a dedicated environment for such tests.

In short, as of now, these things are not decided. So, we continue to replace the tests, but without GitHub Actions Workflow.

rina23q avatar Aug 04 '22 15:08 rina23q

GitHub hosted runner cuts internet connection from time to time.

Why is this an issue for these tests?

How? After merge? Nightly?

Why not doing it like every other PR, in the PR workflow? That's what CI is all about, isn't it?

Having to fix mistakes from the day before just because the tests only run once a day is just too much overhead IMO!

matthiasbeyer avatar Aug 04 '22 16:08 matthiasbeyer

GitHub hosted runner cuts internet connection from time to time.

Why is this an issue for these tests? We may get false-negative results because of connection refused!

Why not doing it like every other PR, in the PR workflow? That's what CI is all about, isn't it? This PR is very limited, but we will port more tests. Actually, the whole set of the existing Pysys tests (system tests and end-to-end tests) took 2 hours per one run. That's why I am considering nightly.

Anyway, it is not the scope of this PR. I prefer to have another place to discussion, probably on GitHub Discussions later. I don't have so deep thoughts about this topic yet.

rina23q avatar Aug 04 '22 16:08 rina23q

Requested changes are implemented @rina23q please Review/close this issue

gligorisaev avatar Aug 10 '22 06:08 gligorisaev

call_tedge_config_list.robot file is also added, which is not described above. Is it intended to add this file in this PR?

rina23q avatar Aug 10 '22 12:08 rina23q

call_tedge_config_list.robot file is also added, which is not described above. Is it intended to add this file in this PR?

Corrected, should be part of this PR

gligorisaev avatar Aug 30 '22 12:08 gligorisaev

Why was this merged? It is not even integrated in the CI pipelines yet! Or am I missing something?

matthiasbeyer avatar Aug 30 '22 14:08 matthiasbeyer