coveragepy
coveragepy copied to clipboard
nedbat
IndexError: list index out of range on coverage lcov
Describe the bug
I'm running coverage lcov and getting the following error: IndexError: list index out of range
To Reproduce How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:
- What version of Python are you using?
3.10.4 - What version of coverage.py shows the problem? The output of
coverage debug sysis helpful.
-- sys -------------------------------------------------------
coverage_version: 6.3.2
coverage_module: /Users/joshlincoln/Documents/GitHub/income-detection-service/venv/lib/python3.10/site-packages/coverage/__init__.py
tracer: -none-
CTracer: available
plugins.file_tracers: -none-
plugins.configurers: -none-
plugins.context_switchers: -none-
configs_attempted: .coveragerc
configs_read: /Users/joshlincoln/Documents/GitHub/income-detection-service/.coveragerc
config_file: /Users/joshlincoln/Documents/GitHub/income-detection-service/.coveragerc
config_contents: b'[run]\n include = app/*\n omit = test/*\n branch=true\n[report]\n skip_empty=true\n'
data_file: /Users/joshlincoln/Documents/GitHub/income-detection-service/.coverage
python: 3.10.4 (v3.10.4:9d38120e33, Mar 23 2022, 17:29:05) [Clang 13.0.0 (clang-1300.0.29.30)]
platform: macOS-11.6.5-x86_64-i386-64bit
implementation: CPython
executable: /Users/joshlincoln/Documents/GitHub/income-detection-service/venv/bin/python3
def_encoding: utf-8
fs_encoding: utf-8
pid: 2768
cwd: /Users/joshlincoln/Documents/GitHub/income-detection-service
path:
/Users/joshlincoln/Documents/GitHub/income-detection-service/venv/bin
/Library/Frameworks/Python.framework/Versions/3.10/lib/python310.zip
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload
/Users/joshlincoln/Documents/GitHub/income-detection-service/venv/lib/python3.10/site-packages
environment: HOME = /Users/joshlincoln
command_line: /Users/joshlincoln/Documents/GitHub/income-detection-service/venv/bin/coverage lcov --debug sys
sqlite3_version: 2.6.0
sqlite3_sqlite_version: 3.37.2
sqlite3_temp_store: 0
sqlite3_compile_options: ATOMIC_INTRINSICS=1; COMPILER=clang-13.0.0; DEFAULT_AUTOVACUUM
DEFAULT_CACHE_SIZE=-2000; DEFAULT_FILE_FORMAT=4; DEFAULT_JOURNAL_SIZE_LIMIT=-1
DEFAULT_MMAP_SIZE=0; DEFAULT_PAGE_SIZE=4096; DEFAULT_PCACHE_INITSZ=20
DEFAULT_RECURSIVE_TRIGGERS; DEFAULT_SECTOR_SIZE=4096; DEFAULT_SYNCHRONOUS=2
DEFAULT_WAL_AUTOCHECKPOINT=1000; DEFAULT_WAL_SYNCHRONOUS=2; DEFAULT_WORKER_THREADS=0
ENABLE_FTS3; ENABLE_FTS3_PARENTHESIS; ENABLE_FTS4
ENABLE_FTS5; ENABLE_GEOPOLY; ENABLE_JSON1
ENABLE_MATH_FUNCTIONS; ENABLE_RTREE; MALLOC_SOFT_LIMIT=1024
MAX_ATTACHED=10; MAX_COLUMN=2000; MAX_COMPOUND_SELECT=500
MAX_DEFAULT_PAGE_SIZE=8192; MAX_EXPR_DEPTH=1000; MAX_FUNCTION_ARG=127
MAX_LENGTH=1000000000; MAX_LIKE_PATTERN_LENGTH=50000; MAX_MMAP_SIZE=0x7fff0000
MAX_PAGE_COUNT=1073741823; MAX_PAGE_SIZE=65536; MAX_SQL_LENGTH=1000000000
MAX_TRIGGER_DEPTH=1000; MAX_VARIABLE_NUMBER=32766; MAX_VDBE_OP=250000000
MAX_WORKER_THREADS=8; MUTEX_PTHREADS; OMIT_AUTOINIT
SYSTEM_MALLOC; TCL; TEMP_STORE=1
THREADSAFE=1
-- end -------------------------------------------------------
Wrote LCOV report to coverage.lcov
Traceback (most recent call last):
File "/Users/joshlincoln/Documents/GitHub/income-detection-service/venv/bin/coverage", line 8, in <module>
sys.exit(main())
File "/Users/joshlincoln/Documents/GitHub/income-detection-service/venv/lib/python3.10/site-packages/coverage/cmdline.py", line 945, in main
status = CoverageScript().command_line(argv)
File "/Users/joshlincoln/Documents/GitHub/income-detection-service/venv/lib/python3.10/site-packages/coverage/cmdline.py", line 722, in command_line
total = self.coverage.lcov_report(
File "/Users/joshlincoln/Documents/GitHub/income-detection-service/venv/lib/python3.10/site-packages/coverage/control.py", line 1094, in lcov_report
return render_report(self.config.lcov_output, LcovReporter(self), morfs, self._message)
File "/Users/joshlincoln/Documents/GitHub/income-detection-service/venv/lib/python3.10/site-packages/coverage/report.py", line 35, in render_report
return reporter.report(morfs, outfile=outfile)
File "/Users/joshlincoln/Documents/GitHub/income-detection-service/venv/lib/python3.10/site-packages/coverage/lcovreport.py", line 35, in report
self.get_lcov(fr, analysis, outfile)
File "/Users/joshlincoln/Documents/GitHub/income-detection-service/venv/lib/python3.10/site-packages/coverage/lcovreport.py", line 57, in get_lcov
line = source_lines[covered-1].encode("utf-8")
IndexError: list index out of range
- What versions of what packages do you have installed? The output of
pip freezeis helpful
aiomysql==0.1.0
alembic==1.7.7
anyio==3.5.0
asgiref==3.5.0
attrs==21.4.0
black==22.3.0
cachetools==5.0.0
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.12
click==8.1.2
coverage==6.3.2
cryptography==36.0.2
ddtrace==0.60.1
fastapi==0.74.1
flake8==4.0.1
google-api-core==2.7.1
google-auth==2.6.2
google-cloud-secret-manager==2.9.2
googleapis-common-protos==1.56.0
greenlet==1.1.2
grpc-google-iam-v1==0.12.3
grpcio==1.45.0
grpcio-status==1.45.0
h11==0.12.0
httpcore==0.14.7
httptools==0.3.0
httpx==0.22.0
idna==3.3
iniconfig==1.1.1
isort==5.10.1
Mako==1.2.0
MarkupSafe==2.1.1
mccabe==0.6.1
mypy-extensions==0.4.3
numpy==1.22.2
packaging==21.3
pandas==1.4.1
pathspec==0.9.0
pep517==0.12.0
pip-tools==6.6.0
platformdirs==2.5.1
pluggy==1.0.0
proto-plus==1.20.3
protobuf==3.20.0
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.8.0
pycparser==2.21
pydantic==1.9.0
pyflakes==2.4.0
PyMySQL==1.0.2
pyparsing==3.0.7
pytest==7.0.1
pytest-asyncio==0.18.3
pytest-cov==3.0.0
pytest-env==0.6.2
pytest-mock==3.7.0
python-configuration==0.8.2
python-dateutil==2.8.2
python-dotenv==0.20.0
pytz==2022.1
PyYAML==6.0
requests==2.27.1
rfc3986==1.5.0
rsa==4.8
six==1.16.0
sniffio==1.2.0
SQLAlchemy==1.4.32
starlette==0.17.1
tenacity==8.0.0
tomli==2.0.1
typing_extensions==4.1.1
urllib3==1.26.9
uvicorn==0.17.5
uvloop==0.16.0
watchgod==0.8.2
websockets==10.2
- What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you've already worked around the problem, please provide a commit before that fix. I had it print the lines that it's hitting the index error on, it's consistently this code:
['import datetime', 'from typing import List, Optional', '', 'from sqlalchemy.ext.asyncio import AsyncSession', '', 'from app import run_config', 'from app.db.repositories.payroll_assessment import PayrollAssessmentRepository', 'from app.db.tables.next_payroll_date import NextPayrollDate', 'from app.db.tables.payroll_assessment import PayrollGroupAssessment', 'from app.db.tables.payroll_type import PayrollType', 'from app.domain.bank_transaction_provider import BankTransactionProvider', 'from app.domain.income import Income', 'from app.models.schema.payroll_assessment import (', ' InPayrollAssessment,', ' PayrollAssessmentSchema,', ')', 'from app.typings.income import IncomeParams', '', '', 'class IncomeDetection:', ' def __init__(self, session: AsyncSession):', ' self.payroll_group_repository = PayrollAssessmentRepository(session)', ' self.config = run_config.load_config()', '', ' async def detect_income(self, params: IncomeParams) -> Optional[Income]:', ' bt = await BankTransactionProvider.get_user_bank_transactions(params)', ' if len(bt) == 0:', ' return None', ' return Income(transaction_data=bt, assessment_date=params.assessment_date)', '', ' async def upsert_income(', ' self, params: IncomeParams, income: Income', ' ) -> PayrollAssessmentSchema:', ' payload = InPayrollAssessment(', ' user_id=params.user_id,', ' bank_account_id=params.bank_account_id,', ' assessment_date_time=params.assessment_date_time,', ' num_valid_payroll_groups=income.valid_payroll_count,', ' last_30d_income=income.last_30days_income_amount,', ' version=self.config.income_detection_version,', ' payroll_assessment_groups=self.__get_payroll_group_assessments(', ' income.valid_payrolls', ' ),', ' )', ' return await self.payroll_group_repository.create(payload)', '', ' def __get_next_payroll_dates(', ' self, dates: List[datetime.date]', ' ) -> List[NextPayrollDate]:', ' next_payroll_dates = []', ' for date in dates:', ' next_payroll_dates.append(NextPayrollDate(payroll_date=date))', ' return next_payroll_dates', '', ' def __get_payroll_group_assessments(', ' self,', ' payroll_groups,', ' ) -> List[PayrollGroupAssessment]:', ' payroll_group_list = []', ' for payroll in payroll_groups:', ' payroll_group_list.append(', ' PayrollGroupAssessment(', ' payroll_name=payroll.payroll_name,', ' status=payroll.status,', ' confidence_score=payroll.confidence_score,', ' payroll_frequency=payroll.payroll_frequence,', ' is_freq_adjusted=payroll.is_freq_adjusted,', ' frequency=payroll.payroll_frequence,', ' interval=payroll.interval,', ' last_payroll_date=payroll.last_payroll_date,', ' last_30d_earnings=payroll.last_30d_earnings,', ' next_payroll_amount=payroll.next_payroll_amount,', ' estimated_monthly_payroll_amount=(', ' payroll.estimated_monthly_payroll_amount', ' ),', ' payroll_type=PayrollType(name=payroll.payroll_type),', ' next_payroll_dates=self.__get_next_payroll_dates(', ' payroll.next_payroll_dates', ' ),', ' )', ' )', ' return payroll_group_list']
- What commands did you run?
pytest --cov=app tests && coverage lcovExpected behavior I expect it to write to the lcov file without error
Additional context if I add this code here I'm able to avoid the error.
if covered - 1 >= len(source_lines):
continue
For some reason analysis.executed is getting a value that is > len(source_lines) from my code.
Thanks. Can you link us to the specific code you were measuring, and how you ran the tests/coverage?
Hi @nedbat . I can't link to the specific code, it's a private repo. But the file that is causing the error is listed above. I was measuring coverage and generating the report using the following command pytest --cov=app tests && coverage lcov
I added some debug statements to print the covered value and the len(source_lines).
Here is the output.
source_lines: 82
Covered:1
Covered:2
Covered:4
Covered:6
Covered:7
Covered:8
Covered:9
Covered:10
Covered:11
Covered:12
Covered:13
Covered:17
Covered:20
Covered:21
Covered:22
Covered:23
Covered:25
Covered:26
Covered:27
Covered:28
Covered:29
Covered:31
Covered:34
Covered:45
Covered:47
Covered:50
Covered:51
Covered:52
Covered:53
Covered:55
Covered:59
Covered:60
Covered:61
Covered:82
Covered:1812
Covered:1822
Covered:1824
Covered:1826
Covered:1855
For some reason, analysisis.executed is getting values that are larger than the source_lines len.
Can you point me to where analysis.executed is set? Might allow me to see why it's analyzing my code incorrectly.
The source_file does in fact have 82 lines, not sure why it's getting
Covered:1812
Covered:1822
Covered:1824
Covered:1826
Covered:1855
values in the analysis.
Other context, this was previously working fine. Only changes here, was this class was broken up into multiple classes in different files, and now the coverage lcov command is breaking.
@nedbat any idea about this? Can we at least handle the invalid index in the loop?
This looks like a problem where line numbers from one frame are misattributed to another frame. I'm not going to fix it by checking the length of the list, because there's no reason it's limited to numbers that are too large. The wrong-frame numbers could be within the valid range of line numbers. We have to figure out why they are happening.
Can you try running coverage with the --timid flag to use the Python trace function? If that produces different results, it could be a clue.