coveragepy
coveragepy copied to clipboard
nedbat
Absolute paths with [run]relative_files
Describe the bug
The [run]relative_files settings makes coverage store relative file paths in the data file. However this is only valid for files in or below the folder from which coverage is executed, i.e. the coverage current working folder.
Absolute paths are still used for files in or below adjacent folders.
To Reproduce Please use the bash and Windows batch files below to reproduce the problem. Note that GitHub prevents attaching batch file.
#~ Cleanup.
rm -rf DEd4sf45
rm -rf .venv
#~ Create a virtual environment.
#~ Same behavior for 3.9-64, 3.10-64 and 3.11-64.
python -m venv .venv
. ./.venv/bin/activate
#~ Same behavior for version 5.5 and 6.5.0 and 7.2.7.
python -m pip install coverage
python --version --version
coverage --version
RUN_JOB () {
#~ Create folders for code under test and test.
mkdir -p $1/src/foo
mkdir -p $1/test
#~ Create environment.
export PYTHONPATH=$PWD/$1/src
#~ Create code under test.
echo 'def foo():' > $1/src/foo/__init__.py
echo ' pass' >> $1/src/foo/__init__.py
#~ Create test.
echo 'import foo' > $1/test/test.py
echo 'foo.foo()' >> $1/test/test.py
#~ Create the coverage configuration.
echo '[run]' > $1/.coveragerc
echo 'relative_files = True' >> $1/.coveragerc
#~ Note that test code is normally not included in coverage measurement.
echo source = $PWD/$1/src,$PWD/$1/test >> $1/.coveragerc
#~ Setting source or include result in the same behavior.
#~ echo include = $PWD/$1/src/foo/__init__.py,$PWD/$1/test/test.py >> $1/.coveragerc
#~ Move in to the test folder, run the test and create the coverage report.
cd $1/test
coverage run --rcfile=../.coveragerc test.py
#~ Note that the path to the code under test is absolute whereas the path to
#~ the test code is relative.
#~ The relative_files setting only applies for code in or below the folder from
#~ which coverage is executed.
#~ Note that relative_files seems to make no difference.
coverage report --rcfile=../.coveragerc
cd -
}
RUN_JOB 3df5grDR
RUN_JOB DEd4sf45
#~ Aggregate the coverage data.
mv 3df5grDR/test/.coverage DEd4sf45/test/.coverage.3df5grDR
mv DEd4sf45/test/.coverage .coverage.DEd4sf45
#~ CI Clears job data.
rm -rf 3df5grDR
#~ Combine the coverage data and create the report.
cd DEd4sf45/test
coverage combine --keep --rcfile=../.coveragerc
#~ Report fails on missing sources.
coverage report --rcfile=../.coveragerc
cd -
deactivate
@ECHO OFF
REM ~ Cleanup.
RMDIR /S /Q DEd4sf45
RMDIR /S /Q .venv
REM ~ Create a virtual environment.
REM ~ Same behavior for 3.9-64, 3.10-64.
py -3.11-64 -m venv .venv
CALL .venv\Scripts\activate
REM ~ Same behavior for version 5.5 and 6.5.0.
python -m pip install coverage
python --version --version
coverage --version
CALL :RUN_JOB 3df5grDR
CALL :RUN_JOB DEd4sf45
REM ~ Aggregate the coverage data.
COPY 3df5grDR\test\.coverage DEd4sf45\test\.coverage.3df5grDR
RENAME DEd4sf45\test\.coverage .coverage.DEd4sf45
REM ~ CI Clears job data.
RMDIR /S /Q 3df5grDR
REM ~ Combine the coverage data and create the report.
PUSHD DEd4sf45\test
coverage combine --keep --rcfile=..\.coveragerc
REM ~ Report fails on missing sources.
coverage report --rcfile=..\.coveragerc
POPD
GOTO:EOF
:RUN_JOB
REM ~ Create folders for code under test and test.
MKDIR %1\src\foo
MKDIR %1\test
REM ~ Create environment.
SET PYTHONPATH=%CD%\%1\src
REM ~ Create code under test.
ECHO def foo(): > %1\src\foo\__init__.py
ECHO pass >> %1\src\foo\__init__.py
REM ~ Create test.
ECHO import foo > %1\test\test.py
ECHO foo.foo() >> %1\test\test.py
REM ~ Create the coverage configuration.
ECHO [run] > %1\.coveragerc
ECHO relative_files = True >> %1\.coveragerc
REM ~ Note that test code is normally not included in coverage measurement.
ECHO source = %CD%\%1\src,%CD%\%1\test >> %1\.coveragerc
REM ~ Setting source or include result in the same behavior.
REM ~ ECHO include = %CD%\%1\src\foo\__init__.py,%CD%\%1\test\test.py >> %1\.coveragerc
REM ~ Move in to the test folder, run the test and create the coverage report.
PUSHD %1\test
coverage run --rcfile=..\.coveragerc test.py
REM ~ Note that the path to the code under test is absolute whereas the path to
REM ~ the test code is relative.
REM ~ The relative_files setting only applies for code in or below the folder from
REM ~ which coverage is executed.
REM ~ Note that relative_files seems to make no difference.
coverage report --rcfile=..\.coveragerc
POPD
GOTO:EOF
Expected behavior coverage must use relative paths for all files, not only files which are in or below the folder from which coverage is executed, i.e. coverage must use .. for relative paths to files which are in or below folders adjacent to the folder from which coverage is executed.
Additional context NA
Thanks for the issue report. It's hard for me to run Windows and I'm not sure if this is Windows-specific. You said it happens in coverage 5.5 and 6.5.0. Can you try it with the latest version of coverage?
Thanks for the issue report. It's hard for me to run Windows and I'm not sure if this is Windows-specific. You said it happens in coverage 5.5 and 6.5.0. Can you try it with the latest version of coverage?
I will prepare a bash script (I run Debian). The script runs with the latest coverage version (currently 7.2.7). So it most likely happens for all 5.5+ versions.
I can reproduce this behaviour on the latest master branch with Python 3.12.3. With that being said, this may be intentional behaviour that's just lightly documented.
When you set relative_paths = true, the relative paths are computed to be relative to the current running directory. Likely to prevent wild relative paths from system files, or possibly to prevent breaking out from the current directory, the relative paths are only computed for files contained in the same directory or a subdirectory from the current one. All other paths remain absolute paths, which is what is being reported here.
It's not clear if this should be changed to be relative from the .coveragerc instead, which would both allow for the relative paths to have a safe, stable base path but also would allow for that base path to be different than the currently running path. Since source can be specified within the .coveragerc to be relative, it may make sense to establish that as the universal baseline for relative paths.
Example reproduction environment (simplified):
# /env/foo/__init__.py
def foo():
pass
# /env/test/test.py
import foo
foo.foo()
# /env/.coveragerc
[run]
relative_files = true
source = ./src,./test
Commands to run to reproduce:
cd /env/test
python -m coverage run --rcfile=../.coveragerc test.py
python -m coverage report --rcfile=../.coveragerc
Actual output:
Name Stmts Miss Cover
------------------------------------------------------------------------------------
/env/foo/__init__.py 2 0 100%
test.py 2 0 100%
------------------------------------------------------------------------------------
TOTAL 4 0 100%
Expected output:
Name Stmts Miss Cover
------------------------------------------------------------------------------------
../foo/__init__.py 2 0 100%
test.py 2 0 100%
------------------------------------------------------------------------------------
TOTAL 4 0 100%
@owillebo Can you say more about your situation? Why are you running from a directory below the .coveragerc file? Most people run their tests from the root of their project so that all of the files are stored as relative.
I'm reluctant to change the logic of how relative directories are used, since it could change the behavior of other people's configurations.
We have the test code adjacent to the code, i.e. 2 folders holding the code and tests in the root.
root
code
tests
The test framework changes directory to the folder holding a test before running the test so that the test can easily access test data in the test folder.
This results in absolute paths for all code as the code is in a folder adjacent to the folder we run the tests from (i.e. the test folder).
We tried to solve this by moving the .coveragerc file in to the root folder, however the paths remained relative as the current working folder is used instead of the folder holding the .coveragerc file.
My use case is a monorepo where we have separate sub-directories for client, server, and tests. Each of the sub-directories has its own pyproject.toml with its own set of dependencies, virtual env, and tool configuration / scripts.
repo/
├─ client/
│ ├─ .venv/
│ └─ pyproject.toml
├─ server/
│ ├─ .venv/
│ └─ pyproject.toml
└─ tests/
├─ .venv/
└─ pyproject.toml
We want to generate a single report with the whole repository's code coverage (e.g. for visualization in Gitlab merge requests). In CI, we have several test jobs: unit tests for client, unit tests for server, and our integration tests defined in tests/. The integration test job runs the server in a background process, and then runs the tests from tests/.
Our preferred configuration would be to just specify something like COVERAGE_ROOT=".." so that the coverage data uses paths relative to repo/, similar to how we invoke pytest with --rootdir=.. so that its generated JUnit test reports use the full repository path to each test file.
But we did find a workaround: in each test run script, cd .. before running the tests or starting the server. For example, for starting the server with coverage turned on (note that the current working directory always starts in a sub-directory because of the monorepo setup):
# Original script
start-server:
.venv/bin/python -m uvicorn main:app --port 9000
# Workaround with coverage.py
start-server:
cd ../ && \
COVERAGE_FILE=server/.coverage \
server/.venv/bin/python -m coverage run -m uvicorn --app-dir server main:app --port 9000
Also note that you might have to modify your code to work with a variable current working directory, as we did, since the working directory in production will differ from the working directory in test mode.
