talawa-api
talawa-api copied to clipboard
PalisadoesFoundation
Test run time is too long.
Describe the bug
- Test run time is too long.
- They are taking 20 minutes to run
- This is unacceptable
To Reproduce
- Submit a Pull Request
- Tests take too long
Expected behavior
- Much faster execution
Actual behavior
- See above
Screenshots
- N/A.
Additional details
- N/A
Potential internship candidates
Please read this if you are planning to apply for a Palisadoes Foundation internship
- https://github.com/PalisadoesFoundation/talawa/issues/359
This issue did not get any activity in the past 10 days and will be closed in 180 days if no update occurs. Please check if the develop branch has fixed it and report again or close the issue.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
What can be the approach to solve this? @palisadoes ? What can be the possible reasons for the delay of the tests? I have faced this myself but the solution isnt clear to me. @lokeshwar777 How are you gonna solve this?
All i found was this article here : Improve test time And a github discussion : Link There is some use of "threads" as vitest is multi threaded.
Me and Peter discussed about the possible approach. Refer in talawa-projects Slack channel https://thepalisadoes-dyb6419.slack.com/archives/C06NRV8J9A9/p1713548954306069?thread_ts=1713548954.306069&cid=C06NRV8J9A9
@varshith257
- Please propose a directory structure to handle it.
- How can we make the tests in each folder run concurrently?
You proposed splitting the runs into two parallel streams. That would mean hopefully a halving of our 20 minute run time. That's still too long.
@palisadoes Would running the tests for only the files that have been changed work?
@palisadoes Didn't check on that, will research on this and let you know.
@palisadoes I couldn't really find anything about this. We wouldn't know what problems, implementing this might bring unless we do it. Can I work on this (with a mentor if possible) ?
It's not a good approach and never seen in other orgs of testing changed files only. It makes no sense of testing changed files only. How can we rectify that distrupts other files that has dependency if changed files only tetsed.
Possibile method is making tests files subfolders and adding CI workflow for them to test. If adding two subfolders doesn't meet the less time requirement we need to look for more split of tests into subfolders and triger test workflow for them.
From my observation there are few files that are taking most of the time for testing. Should be focused on them to split.
@varshith257 I was also thinking about it.
How can we rectify that distrupts other files that has dependency if changed files only tetsed.
I found out this in the vitest docs, not sure if it helps
https://vitest.dev/config/#maxconcurrency
Ok! Give it a try by creating a small subfolder with 10-15 tests in a folder, using of maxConcurrency variable and check if it works for running tests concurrently. It can be tested locally with a change of pointing directory in the package.json
@karthxk07 Ask here to get assigned this issue
Runtime without maxConcurrency flag(default maxConcurrency =5)
Runtime with maxConcurrency=10 and sequence.concurrent flag
sequence.concurrent was surely running multiple tests at the same time, but most of them threw errors . This might be because tests are written in sequential order, which have to be run in a particular order only. We might have to scrape this approach. @varshith257 how should i proceed further.
Would it make sense to reload test data for every test file to reduce the risk?
Karthik, Follow the approach I proposed of making subdirectories. Try to split the tests based on the number of tests in each test file and try to equalize them.Start with 2 and let's see how much it reduces time and will further optimise the approach based on it.
@palisadoes @varshith257 If these tests share dependencies or resources, running them concurrently could lead to race conditions, flakiness, or incorrect results due to interference.
Instead, it might be more beneficial to focus on optimizing the test execution strategy within a single process or thread, ensuring that tests with dependencies or shared resources are executed sequentially or in a controlled manner to prevent interference.
@palisadoes I think we should focus on fine-tuning pool options. ,
Currently we use "test": "vitest run --pool forks --poolOptions.forks.singleFork --coverage",,
I tried using other options, but some test cases failed.
@palisadoes and resources for this are very limited, nothing clearing explains, it depends on different codebases.
I saw this in a devops newsletter
Large monorepos can often pose a challenge for traditional CI systems, leading to slow build times. This post looks at using Yarn to only run tests against components that have changed, speeding up feedback.
https://mathieularose.com/ci-cd-pipeline-change-based-testing-yarn-based-monorepo
@palisadoes How can we rectify that disrupts other files that have dependencies if changed files are only tested?
I just saw the link to it and wondered whether it would be useful.
Do you think the selective parallelism method is the best approach for us? If so, then go ahead.
@palisadoes I am trying to run parallel tests for Mutation and Query because these will not be dependent on each other. I am just figuring out how I can do this with vitest.
@palisadoes I am trying to implement change-based testing and will run the test for dependent files.
@palisadoes I tried implementing separate workflows for mutation and query but didn't succeed because of libraries, middlewares, and global dependencies.
Then my observation was that most of the time goes into running test cases of mutation; most of the files have more than 15 test cases; some even have 25 test cases, which leads to excessive time. For that, I tried running test cases in parallel using the vitest inbuilt functionality, i.e., running-tests-concurrently but a lot of test cases were failing.