Collect coverage for live tests

[AlanCoding]

SUMMARY

Obligatory gripe:

This doesn't collect server-side coverage, which is the key mechanistic thing happening here. This only collects coverage from the pytest process, which is superficial, client-side stuff in most cases. It might go a little deeper on how the test is written, but totally case-by-case. I generally expect the interesting stuff happens via background tasks in the dispatcher.

ISSUE TYPE

• Bug, Docs Fix or other nominal change

COMPONENT NAME

• API

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 75.49%. Comparing base (d6482d3) to head (9a37272).

:white_check_mark: All tests successful. No failed tests found.

:rocket: New features to boost your workflow:

• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[AlanCoding]

didn't work

[AlanCoding]

some clippings

It looks like we're saving the same xml file. But that's possibly not was expected?? What is expected?

api-test

make test PYTEST_ADDOPTS="--create-db --cov --cov-report=xml --junitxml=reports/junit.xml"
make[1]: Entering directory '/awx_devel'
if [ "/var/lib/awx/venv" ]; then \
	. /var/lib/awx/venv/awx/bin/activate; \
fi; \
PYTHONDONTWRITEBYTECODE=1 py.test -p no:cacheprovider -n auto awx/main/tests/unit awx/main/tests/functional awx/conf/tests
...
--------------- generated xml file: /awx_devel/reports/junit.xml ---------------
...
--------------- generated xml file: /awx_devel/awxkit/report.xml ---------------

Ran:

• Upload test coverage to Codecov
• Upload test results to Codecov

Did not run:

• Upload awx jUnit test reports

dev-env

Run docker exec tools_awx_1 /bin/bash -c "PYTEST_ADDOPTS='--cov=awx --cov-report=xml --junitxml=../../../../reports/junit.xml --no-cov-on-fail' make live_test"
cd awx/main/tests/live && py.test tests/
...
--------------- generated xml file: /awx_devel/reports/junit.xml ---------------
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.11.11-final-0 _______________

Coverage XML written to file coverage.xml
================== 22 passed, 3 warnings in 277.91s (0:04:37) ==================

Did not run:

• Upload test coverage to Codecov
• Upload test results to Codecov

[AlanCoding]

@webknjaz I'm trying to understand why the coverage would skip

https://github.com/ansible/awx/blob/3db2e04efe00c8fd0265de1f2a1f7388da95fd34/.github/workflows/ci.yml#L67-L69

So my current status is that I don't know where steps.make-run.outputs.cov-report-files comes from. This produces /awx_devel/reports/junit.xml. Is that the same thing or not the same thing? How does this get connected up?

[webknjaz]

It's something generic I had in tox-based projects. I reproduced it in make for now: https://github.com/ansible/awx/blob/3db2e04efe00c8fd0265de1f2a1f7388da95fd34/Makefile#L326-L330

[AlanCoding]

Still giving the same result as my screenshot above.

I just don't know what specific artifacts it needs. Perhaps it's reports/coverage.xml that it's missing?

[webknjaz]

@AlanCoding could you restart that job in debug mode?

One artifact is a JUnit XML file while the other is Cobertura XML. They should be host paths, that's where the Codecov uploader runs. And the source code should be extracted with the same relative layout, I imagine.

My hunch is that maybe the env var evaluates to an empty string because the make target runs in a container. Could you also check that $GITHUB_ACTIONS is bypassed into the container?

[AlanCoding]

with debug mode

https://github.com/ansible/awx/actions/runs/15690322675/job/44251182441

[webknjaz]

@AlanCoding that confirms my suspicion that the GITHUB_ACTIONS env var isn't exposed to the container. And so when you run make in such an isolation, it doesn't set the outputs since it thinks that it's not a CI run. The solution would be to actually pass that env var in. This could be in the initial container invocation, although I don't know how difficult it is to plug into docker-compose or it could be done in the step but then it wouldn't be generic.

[sonarqubecloud[bot]]

Please retry analysis of this Pull-Request directly on SonarQube Cloud