Make Unit Test Great!

[HuSharp]

Development Task

• The unstable CI has been a hinder to our daily development. Sometimes we have to run it again and again, and again ... to merge a PR.
• When run make basic-test in local machine, there will be a lot of strange problems, as well as running extremely slow :(

Need 5:44.15 total for Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GH, 144 core 72 CPU but local mac(m1pro 16G) needs > 15min

We use a wrong way to parallel. Some test cases are not side effect free, for example:

• the enable and disable of the failpoint
• modification of the configuration
• change of the global variable in a single process. etc...

What's the solution?

Add back parallel by run a single test case in a isolated OS process #7952

If the test cases with side effect run parallelly with others, it could cause some test fail unexpectedly. When they're in a single OS process, make them parallel is not a big deal.

print function time

when using ./pd-ut run --junitfile xxx, can get junitfile as XML file to get every function cost time.

Furthermore

• handling the coverage.txt & integration with the CI script #7991
• classify some real integrated tests and move them to the tests directory #7930
• support cache mechanism to accelerate bin build #8069
• change required ci https://github.com/ti-community-infra/configs/pull/1048
• support running integration test #8200
• support parallel parameter #8186
• unify tests HTTP client #8199
• support an alloc server to alloc URL #8203

Related

• Ref TiDB https://github.com/pingcap/tidb/issues/30822
• Enhance the integration test of CI https://github.com/tikv/pd/issues/7298

Result Show

Local Test

Since we run parallel now, the CI runs faster than before, on my server computer (Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GH, 144 core 72 CPU): reduce from 6min -> 1min

for make ut:

time make ut
building task finish, parallelism=144, count=641, takes=10.154869881s
run all tasks takes 50.559951196s
> make ut  327.15s user 48.61s system 615% cpu 1:01.03 total

for make basic-test

time make basic-test
CGO_ENABLED=1 go test -tags tso_function_test,deadlock -timeout 20m -race
> make basic-test 1176.46s user 169.41s system 473% cpu 5:44.15 total

CI Result

Before	After