openvino
openvino copied to clipboard
openvinotoolkit
[Good First Issue]: Enable parallel execution of OVC Python API python tests
Context
Current CI (https://github.com/openvinotoolkit/openvino/blob/c0381ab08d2fcdc51c621196e935aa02c64a4b50/.github/workflows/linux.yml#L869) runs all the python tests sequentially one by one and it takes too much time. The idea is to run all those tests in parallel by using pytest-xdist plugin, but tests should be modified for that. Please find bellow the common tests issues:
- Random inputs generation as parametrization arguments. Solution: all random test arguments should be generated inside of the tests
- Tests create, use and remove the same artifacts (temporary data, models, etc). Hot fix solution: all test data should be generated in the system temporary directory (TMPDIR) which can be unique for each thread. Long term solution: all test data should be generated in the unique temporary directory or with unique names.
What needs to be done?
- Modify OVC Python API tests to solve issues described in the Context. Tests are located here - https://github.com/openvinotoolkit/openvino/tree/master/tests/layer_tests/ovc_python_api_tests
- Enable parallel execution of those test in the CI (https://github.com/openvinotoolkit/openvino/blob/4a0098b26ad6c4cbae97fbe34505fa63c08036eb/.github/workflows/linux.yml#L1001C15-L1001C35)
Example Pull Requests
https://github.com/openvinotoolkit/openvino/pull/20613
Resources
- Contribution guide - start here!
- Intel DevHub Discord channel - engage in discussions, ask questions and talk to OpenVINO developers
- What is OpenVINO?
- Blog post on contributing to OpenVINO
- User documentation
Contact points
@mryzhov @akashchi
Ticket
No response
I would be happy to help with the issue. Could you assign this issue to me?
Hello @boringlyBoring, is there anything we can help you with? Do you have any questions?
The task has been returned to be picked up due to current assignee's inactivity. @boringlyBoring if you're still out there working on it, please let us know - we'll return the task to you. :)
Hi! I'd like to work on this issue. I have a couple of questions.
- For the temporary directory issue, isn't the
temp_dirfixture here already generating a unique temporary directory per test? - I couldn't figure out where the random test inputs are being used. Can you link an example test taking random arguments?
Hi @Prakhar314, I would recommend you to run those tests in parallel as first step and collect list of the failed tests. It may help to find out the root cause. tmp_dir issue could not be applicable for those tests.
Hey @mryzhov i would like work on this issue. could you please assign me this issue?
@mlukasze thanks for assigning me this issue ill work on it. and create pr as soon as possible
hey @mlukasze @mryzhov
I wanted to provide an update regarding the issue assigned to me. I've been diligently working on it for the past week, but unfortunately, progress has been slower than anticipated due to unforeseen personal commitments. Specifically, I have exams scheduled that require my immediate attention and focus. I acknowledge that it may take some time for me to resolve this issue. I would appreciate your patience as I navigate through my exams and allocate time to address the problem effectively. I am committed to contributing to the project and will do my best to resolve this issue as soon as my schedule permits.
Hello @Ruman-12, no worries! As long as you let us know that it's in progress it's alright. :)
Hello @SaltTheClouds, thanks for taking a look! Please let us know if you have any questions, we're here to help. :)
Hello @SaltTheClouds, thanks for taking a look! Please let us know if you have any questions, we're here to help. :)
Thank you for including me
Hello @SaltTheClouds, thanks for taking a look! Please let us know if you have any questions, we're here to help. :)
I just wanted to check in with you. I'm still working on this issue, but I'm very new to this and I had some technical problems that I didn't expect. I believe I have them worked out now though.
Thanks for being interested in this issue. It looks like this ticket is already assigned to a contributor. Please communicate with the assigned contributor to confirm the status of the issue.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hey folks, I would like to contribute, could I take a look at it? Seems interesting !!!
@SaltTheClouds do not worry, maybe some other Good First Issue will fit to you better? For now I'm reassigning task to @mahajanparth
@mlukasze . Just wanted to confirm some things , even with no parallelism , i can see there are 8 test cases failing and a few skipped. Are there any additional requirements that i am failing to install or is it okay for now ?
Program : test_tf_output_order.py .x.x....xxx.xx.x is the culprit
Hey, @mlukasze
- I have converted all the test cases to be run in parallel using pytest-xdist .
- Also shifted random tensor creation from being done in the test data before parameteritization to inside of the test cases itself.
- The temp dir fixture was already there so the second part of the ticket didn't need any attention as per now .
- I am still finding a few test cases to be failing with both sequential and parallelized approach does it need attention from my end ? Please find the running Time difference in the images below :
- parallel execution 8 core : 98 seconds ( 542 passed , 12 skipped , 8 failed )
- sequential execution : 407 seconds ( 542 passed , 12 skipped , 8 failed )
sequential execution :
parallel execution :
should i raise a PR for the changes ?
That sounds like a significant speedup! Could you please open a PR with your changes? It would speed up the review process a lot.