coveragepy icon indicating copy to clipboard operation
coveragepy copied to clipboard

(Python 3.14) 7.11.1-7.11.3 performance 2x slower than 7.11.0

Open MetRonnie opened this issue 5 months ago • 16 comments

Describe the bug Running our tests with coverage 7.11.3 is taking about twice as long as 7.11.0. This only occurs on Python 3.14.

To Reproduce How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Think about the time it will take us to recreate your situation: the easier you make it, the more likely your issue will be addressed.

Sorry, only have link to CI for now: https://github.com/MetRonnie/cylc-flow/actions/runs/19274020446/workflow

Answer the questions below:

What version of Python are you using?

3.14.0 (h32b2ec7_102_cp314 conda-forge)

What version of coverage.py shows the problem? The output of coverage debug sys is helpful.

7.11.1+

coverage debug sys output
-- sys -------------------------------------------------------
         coverage_version: 7.11.3
          coverage_module: /home/runner/micromamba/envs/cylc-functional-test/lib/python3.14/site-packages/coverage/__init__.py
                     core: -none-
                  CTracer: available from /home/runner/micromamba/envs/cylc-functional-test/lib/python3.14/site-packages/coverage/tracer.cpython-314-x86_64-linux-gnu.so
     plugins.file_tracers: -none-
      plugins.configurers: -none-
plugins.context_switchers: -none-
        configs_attempted: /home/runner/work/cylc-flow/cylc-flow/.coveragerc
             configs_read: /home/runner/work/cylc-flow/cylc-flow/.coveragerc
              config_file: /home/runner/work/cylc-flow/cylc-flow/.coveragerc
          config_contents: b"# This is the Coverage.py configuration file. This is used by CI when running\n# the tests and collecting coverage\n\n[run]\nbranch = True\ncover_pylib = False\nconcurrency = thread\ndata_file = .coverage\ndisable_warnings =\n    trace-changed\n    module-not-python\n    module-not-imported\n    no-data-collected\n    module-not-measured\nomit =\n    tests/*\n    */cylc/flow/*_pb2.py\n    cylc/flow/etc/*\n    cylc/flow/scripts/report_timings.py\nparallel = True\nsource = ./cylc\ntimid = False\n\n[report]\nexclude_lines =\n    pragma: no cover\n\n    # Don't complain if tests don't hit defensive assertion code:\n    raise NotImplementedError\n    return NotImplemented\n\n    # Ignore type checking code:\n    if (typing\\.)?TYPE_CHECKING:\n    @overload( |$)\n\n    # Don't complain about ellipsis (exception classes, typing overloads etc):\n    \\.\\.\\.\n\n    # Ignore abstract methods\n    @(abc\\.)?abstractmethod\n\nfail_under=0\nignore_errors = False\nomit =\n    tests/*\n 
                data_file: -none-
                   python: 3.14.0 | packaged by conda-forge | (main, Oct 22 2025, 23:24:08) [GCC 14.3.0]
                 platform: Linux-6.11.0-1018-azure-x86_64-with-glibc2.39
           implementation: CPython
                    build: main
                           Oct 22 2025 23:24:08
              gil_enabled: True
               executable: /home/runner/micromamba/envs/cylc-functional-test/bin/python3.14
             def_encoding: utf-8
              fs_encoding: utf-8
                      pid: 2819
                      cwd: /home/runner/work/cylc-flow/cylc-flow
                     path: /home/runner/micromamba/envs/cylc-functional-test/bin
                           /home/runner/micromamba/envs/cylc-functional-test/lib/python314.zip
                           /home/runner/micromamba/envs/cylc-functional-test/lib/python3.14
                           /home/runner/micromamba/envs/cylc-functional-test/lib/python3.14/lib-dynload
                           /home/runner/micromamba/envs/cylc-functional-test/lib/python3.14/site-packages
                           __editable__.cylc_flow-8.7.0.dev0.finder.__path_hook__
              environment: CYLC_COVERAGE = 1
                           HOME = /home/runner
             command_line: /home/runner/micromamba/envs/cylc-functional-test/bin/coverage debug sys
                     time: 2025-11-11 18:02:14

What versions of what packages do you have installed? The output of pip freeze is helpful.

pip freeze output
aiosmtpd==1.4.6
ansimarkup==2.1.0
async-generator==1.10
atpublic==6.0.2
attrs==25.4.0
bandit==1.8.6
certifi==2025.10.5
charset-normalizer==3.4.4
classify-imports==4.2.0
click==8.3.0
colorama==0.4.6
contourpy==1.3.3
coverage==7.11.3
cycler==0.12.1
-e git+https://github.com/MetRonnie/cylc-flow@89989f5619ebd709222c62cebfa938c9173ce59d#egg=cylc_flow
execnet==2.1.1
flake8==7.3.0
flake8-broken-line==1.0.0
flake8-bugbear==25.10.21
flake8-builtins==3.1.0
flake8-comprehensions==3.17.0
flake8-debugger==4.1.2
flake8-implicit-str-concat==0.6.0
flake8-mutable==1.2.0
flake8-type-checking==3.0.0
fonttools==4.60.1
graphene==3.4.3
graphql-core==3.2.7
graphql-relay==3.2.0
idna==3.11
iniconfig==2.3.0
Jinja2==3.0.3
kiwisolver==1.4.9
markdown-it-py==4.0.0
MarkupSafe==3.0.3
matplotlib==3.10.7
mccabe==0.7.0
mdurl==0.1.2
metomi-isodatetime==1!3.1.0
mypy==1.18.2
mypy_extensions==1.1.0
numpy==2.3.4
packaging==25.0
pathspec==0.12.1
pillow==12.0.0
pluggy==1.6.0
protobuf==6.33.0
psutil==7.1.3
pycodestyle==2.14.0
pyflakes==3.4.0
Pygments==2.19.2
Pympler==1.1
pyparsing==3.2.5
pytest==9.0.0
pytest-asyncio==1.3.0
pytest-cov==7.0.0
pytest-mock==3.15.1
pytest-xdist==3.8.0
python-dateutil==2.9.0.post0
PyYAML==6.0.3
pyzmq==27.1.0
requests==2.32.5
rich==14.2.0
six==1.17.0
sqlparse==0.5.3
stevedore==5.5.0
testfixtures==10.0.0
towncrier==25.8.0
types-Jinja2==2.11.9
types-MarkupSafe==1.1.10
types-protobuf==6.32.1.20251105
typing_extensions==4.15.0
urllib3==2.5.0
urwid==3.0.3
wcwidth==0.2.14

What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you've already worked around the problem, please provide a commit before that fix.

https://github.com/cylc/cylc-flow/tree/2d33667a78ee256b3d263ad3dbeb1d65a3019212

What commands should we run to reproduce the problem? Be specific. Include everything, even git clone, pip install, and so on. Explain like we're five!

Steps listed in https://github.com/MetRonnie/cylc-flow/actions/runs/19274020446/workflow

Expected behavior Performance should not be slower than 7.11.0. From the release notes, it should be faster, if anything.

Additional context

https://github.com/MetRonnie/cylc-flow/actions/runs/19274020446/usage

Image

MetRonnie avatar Nov 11 '25 18:11 MetRonnie

Those numbers are concerning. Can you add --debug=sys,core to your coverage runs so we can double-check that the right core is being used?

nedbat avatar Nov 11 '25 18:11 nedbat

I tested this with 7.11.3 vs 7.11.0. https://github.com/MetRonnie/cylc-flow/actions/runs/19297527499

Python 3.14

In 7.11.3 only:

in core.py
core.py: core from config is None
core.py: Using sysmon because SYSMON_DEFAULT is set
core.py: Using core=sysmon

In both 7.11.3 and 7.11.0, the sys output contains

core: SysMonitor

Python 3.13

In 7.11.3 only:

in core.py
core.py: core from config is None
core.py: Defaulting to ctrace core
core.py: Using core=ctrace

In 7.11.3 and 7.11.0:

   core: CTracer
CTracer: available from /home/runner/micromamba/envs/cylc-functional-test/lib/python3.13/site-packages/coverage/tracer.cpython-313-x86_64-linux-gnu.so

MetRonnie avatar Nov 12 '25 12:11 MetRonnie

Thanks. I'll have to try some local experiments with this repo to see what's going on. My own timing tests show a slight slowdown, but only ~5%.

nedbat avatar Nov 13 '25 00:11 nedbat

Hi, we're also seeing a significant change in performance for the worse on 7.11.3 compared to 7.11.0 on Python 3.14, both on the GIL and the free-threading build.

This is data from the last run, but we've observed CI with coverage==7.11.3 taking about twice as long consistently:

Python 3.14 w/ coverage==7.11.0: 5m 49s Python 3.14 w/ coverage==7.11.3: 10m 45s

Python 3.14t w/ coverage==7.11.0: 6m 32s Python 3.14t w/ coverage==7.11.3: 12m 3s

Here's some debug info:

With coverage==7.11.3

Installed packages:

py3.14-common: asttokens==3.0.0,attrs==25.4.0,brotli==1.2.0,certifi==2025.11.12,charset-normalizer==3.4.4,colorama==0.4.6,coverage==7.11.3,docker==7.1.0,docopt==0.6.2,executing==2.2.1,h11==0.16.0,h2==4.3.0,hpack==4.1.0,httpcore==1.0.9,hyperframe==6.1.0,idna==3.11,iniconfig==2.3.0,jsonschema==4.25.1,jsonschema-specifications==2025.9.1,MarkupSafe==3.0.3,packaging==25.0,pip==24.0,pluggy==1.6.0,py==1.11.0,Pygments==2.19.2,PySocks==1.7.1,pytest==9.0.1,pytest-asyncio==1.3.0,pytest-cov==7.0.0,pytest-forked==1.6.0,pytest-localserver==0.9.0.post0,pytest-watch==4.2.0,PyYAML==6.0.3,referencing==0.37.0,requests==2.32.5,responses==0.25.8,rpds-py==0.28.0,sentry-sdk @ file:///home/runner/work/sentry-python/sentry-python/.tox/.tmp/package/1/sentry_sdk-2.44.0-0.editable-py2.py3-none-any.whl#sha256=df0ecb704167f9adab412b3df95f7dc443179972aeefbb9f7dd4ed27312f5851,setuptools==80.9.0,socksio==1.0.0,urllib3==2.5.0,watchdog==6.0.0,Werkzeug==3.1.3

coverage debug info:

Combined data file .coverage-sentry-py3.14-common
-- sys -------------------------------------------------------
               coverage_version: 7.11.3
                coverage_module: /opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/site-packages/coverage/__init__.py
                           core: -none-
                        CTracer: available from /opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/site-packages/coverage/tracer.cpython-314-x86_64-linux-gnu.so
           plugins.file_tracers: -none-
            plugins.configurers: -none-
      plugins.context_switchers: -none-
              configs_attempted: /home/runner/work/sentry-python/sentry-python/.coveragerc
                                 /home/runner/work/sentry-python/sentry-python/setup.cfg
                                 /home/runner/work/sentry-python/sentry-python/tox.ini
                                 /home/runner/work/sentry-python/sentry-python/pyproject.toml
                   configs_read: /home/runner/work/sentry-python/sentry-python/tox.ini
                                 /home/runner/work/sentry-python/sentry-python/pyproject.toml
                    config_file: /home/runner/work/sentry-python/sentry-python/pyproject.toml
                config_contents: b'#\n# Tool: Coverage\n#\n\n[tool.coverage.run]\nbranch = true\nomit = [\n    "/tmp/*",\n    "*/tests/*",\n    "*/.venv/*",\n]\n\n[tool.coverage.report]\nexclude_also = [\n    "if TYPE_CHECKING:",\n]\n\n#\n# Tool: Pytest\n#\n\n[tool.pytest.ini_options]\naddopts = "-vvv -rfEs -s --durations=5 --cov=./sentry_sdk --cov-branch --cov-report= --tb=short --junitxml=.junitxml"\nasyncio_mode = "strict"\nasyncio_default_fixture_loop_scope = "function"\nmarkers = [\n    "tests_internal_exceptions: Handle internal exceptions just as the SDK does, to test it. (Otherwise internal exceptions are recorded and reraised.)",\n]\n\n[tool.pytest-watch]\nverbose = true\nnobeep = true\n\n#\n# Tool: Mypy\n#\n\n[tool.mypy]\nallow_redefinition = true\ncheck_untyped_defs = true\ndisallow_any_generics = true\ndisallow_incomplete_defs = true\ndisallow_subclassing_any = true\ndisallow_untyped_decorators = true\ndisallow_untyped_defs = true\nno_implicit_optional = true\npython_version = "3.11"\nstrict_equality = true\nstrict_optional = true\nwarn_redundant_casts = true\nwarn_unused_configs = true\nwarn_unused_ignores = true\n\n# Relaxations for code written before mypy was introduced\n# Do not use wildcards in module paths, otherwise added modules will\n# automatically have the same set of relaxed rules as the rest\n[[tool.mypy.overrides]]\nmodule = "cohere.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "django.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "pyramid.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "psycopg2.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "pytest.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "aiohttp.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "anthropic.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "sanic.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "tornado.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "fakeredis.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "rq.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "pyspark.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "asgiref.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "langchain_core.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "langchain.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "langgraph.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "google.genai.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "executing.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "asttokens.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "pure_eval.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "blinker.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "sentry_sdk._queue"\nignore_missing_imports = true\ndisallow_untyped_defs = false\n\n[[tool.mypy.overrides]]\nmodule = "sentry_sdk._lru_cache"\ndisallow_untyped_defs = false\n\n[[tool.mypy.overrides]]\nmodule = "celery.app.trace"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "flask.signals"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "huey.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "openai.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "openfeature.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "huggingface_hub.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "arq.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "grpc.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "agents.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "dramatiq.*"\nignore_missing_imports = true\n\n#\n# Tool: Ruff (linting and formatting)\n#\n\n[tool.ruff]\n# Target Python 3.7+ (minimum version supported by ruff)\ntarget-version = "py37"\n\n# Exclude files and directories\nextend-exclude = [\n    "*_pb2.py",     # Protocol Buffer files (covers all pb2 files including grpc_test_service_pb2.py)\n    "*_pb2_grpc.py", # Protocol Buffer files (covers all pb2_grpc files including grpc_test_service_pb2_grpc.py)\n    "checkouts",    # From flake8\n    "lol*",         # From flake8\n]\n\n[tool.ruff.lint]\n# Match flake8\'s default rule selection exactly\n# Flake8 by default only enables E and W (pycodestyle) + F (pyflakes)\nselect = [\n    "E",   # pycodestyle errors (same as flake8 default)\n    "W",   # pycodestyle warnings (same as flake8 default)\n    "F",   # Pyflakes (same as flake8 default)\n    # Note: B and N rules are NOT enabled by default in flake8\n    # They were only active through the plugins, which may not have been fully enabled\n]\n\n# Use ONLY the same ignores as the original flake8 config + compatibility for this codebase\nignore = [\n    "E203", # Whitespace before \':\'\n    "E501", # Line too long\n    "E402", # Module level import not at top of file\n    "E731", # Do not assign a lambda expression, use a def\n    "B014", # Redundant exception types\n    "N812", # Lowercase imported as non-lowercase\n    "N804", # First argument of classmethod should be named cls\n\n    # Additional ignores for codebase compatibility\n    "F401", # Unused imports - many in TYPE_CHECKING blocks used for type comments\n    "E721", # Use isinstance instead of type() == - existing pattern in this codebase\n]\n\n[tool.ruff.format]\n# ruff format already excludes the same files as specified in extend-exclude\n# Ensure Python 3.7 compatibility - avoid using Python 3.9+ syntax features\nskip-magic-trailing-comma = false\n'
                      data_file: /home/runner/work/sentry-python/sentry-python/.coverage
                         python: 3.14.0 (main, Oct  7 2025, 13:06:58) [GCC 11.4.0]
                       platform: Linux-6.8.0-1041-azure-x86_64-with-glibc2.35
                 implementation: CPython
                          build: main
                                 Oct  7 2025 13:06:58
                    gil_enabled: True
                     executable: /opt/hostedtoolcache/Python/3.14.0/x64/bin/python
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 24656
                            cwd: /home/runner/work/sentry-python/sentry-python
                           path: 
                                 /opt/hostedtoolcache/Python/3.14.0/x64/bin
                                 /opt/hostedtoolcache/Python/3.14.0/x64/lib/python314.zip
                                 /opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14
                                 /opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/lib-dynload
                                 /opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/site-packages
                    environment: HOME = /home/runner
                   command_line: /opt/hostedtoolcache/Python/3.14.0/x64/bin/coverage xml --debug=sys,core
                           time: 2025-11-13 08:17:25
-- end -------------------------------------------------------

With coverage==7.11.0

Installed packages:

py3.14-common: asttokens==3.0.0,attrs==25.4.0,brotli==1.2.0,certifi==2025.11.12,charset-normalizer==3.4.4,colorama==0.4.6,coverage==7.11.0,docker==7.1.0,docopt==0.6.2,executing==2.2.1,h11==0.16.0,h2==4.3.0,hpack==4.1.0,httpcore==1.0.9,hyperframe==6.1.0,idna==3.11,iniconfig==2.3.0,jsonschema==4.25.1,jsonschema-specifications==2025.9.1,MarkupSafe==3.0.3,packaging==25.0,pip==24.0,pluggy==1.6.0,py==1.11.0,Pygments==2.19.2,PySocks==1.7.1,pytest==9.0.1,pytest-asyncio==1.3.0,pytest-cov==7.0.0,pytest-forked==1.6.0,pytest-localserver==0.9.0.post0,pytest-watch==4.2.0,PyYAML==6.0.3,referencing==0.37.0,requests==2.32.5,responses==0.25.8,rpds-py==0.28.0,sentry-sdk @ file:///home/runner/work/sentry-python/sentry-python/.tox/.tmp/package/1/sentry_sdk-2.44.0-0.editable-py2.py3-none-any.whl#sha256=7c57d2553ee71e47ac192d1b8b55b4f5d49cfbdc18d769d6ba817c46ad3e9614,setuptools==80.9.0,socksio==1.0.0,urllib3==2.5.0,watchdog==6.0.0,Werkzeug==3.1.3

coverage debug info:

-- sys -------------------------------------------------------
               coverage_version: 7.11.0
                coverage_module: /opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/site-packages/coverage/__init__.py
                           core: -none-
                        CTracer: available from /opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/site-packages/coverage/tracer.cpython-314-x86_64-linux-gnu.so
           plugins.file_tracers: -none-
            plugins.configurers: -none-
      plugins.context_switchers: -none-
              configs_attempted: /home/runner/work/sentry-python/sentry-python/.coveragerc
                                 /home/runner/work/sentry-python/sentry-python/setup.cfg
                                 /home/runner/work/sentry-python/sentry-python/tox.ini
                                 /home/runner/work/sentry-python/sentry-python/pyproject.toml
                   configs_read: /home/runner/work/sentry-python/sentry-python/tox.ini
                                 /home/runner/work/sentry-python/sentry-python/pyproject.toml
                    config_file: /home/runner/work/sentry-python/sentry-python/pyproject.toml
                config_contents: b'#\n# Tool: Coverage\n#\n\n[tool.coverage.run]\nbranch = true\nomit = [\n    "/tmp/*",\n    "*/tests/*",\n    "*/.venv/*",\n]\n\n[tool.coverage.report]\nexclude_also = [\n    "if TYPE_CHECKING:",\n]\n\n#\n# Tool: Pytest\n#\n\n[tool.pytest.ini_options]\naddopts = "-vvv -rfEs -s --durations=5 --cov=./sentry_sdk --cov-branch --cov-report= --tb=short --junitxml=.junitxml"\nasyncio_mode = "strict"\nasyncio_default_fixture_loop_scope = "function"\nmarkers = [\n    "tests_internal_exceptions: Handle internal exceptions just as the SDK does, to test it. (Otherwise internal exceptions are recorded and reraised.)",\n]\n\n[tool.pytest-watch]\nverbose = true\nnobeep = true\n\n#\n# Tool: Mypy\n#\n\n[tool.mypy]\nallow_redefinition = true\ncheck_untyped_defs = true\ndisallow_any_generics = true\ndisallow_incomplete_defs = true\ndisallow_subclassing_any = true\ndisallow_untyped_decorators = true\ndisallow_untyped_defs = true\nno_implicit_optional = true\npython_version = "3.11"\nstrict_equality = true\nstrict_optional = true\nwarn_redundant_casts = true\nwarn_unused_configs = true\nwarn_unused_ignores = true\n\n# Relaxations for code written before mypy was introduced\n# Do not use wildcards in module paths, otherwise added modules will\n# automatically have the same set of relaxed rules as the rest\n[[tool.mypy.overrides]]\nmodule = "cohere.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "django.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "pyramid.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "psycopg2.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "pytest.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "aiohttp.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "anthropic.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "sanic.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "tornado.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "fakeredis.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "rq.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "pyspark.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "asgiref.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "langchain_core.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "langchain.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "langgraph.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "google.genai.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "executing.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "asttokens.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "pure_eval.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "blinker.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "sentry_sdk._queue"\nignore_missing_imports = true\ndisallow_untyped_defs = false\n\n[[tool.mypy.overrides]]\nmodule = "sentry_sdk._lru_cache"\ndisallow_untyped_defs = false\n\n[[tool.mypy.overrides]]\nmodule = "celery.app.trace"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "flask.signals"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "huey.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "openai.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "openfeature.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "huggingface_hub.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "arq.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "grpc.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "agents.*"\nignore_missing_imports = true\n\n[[tool.mypy.overrides]]\nmodule = "dramatiq.*"\nignore_missing_imports = true\n\n#\n# Tool: Ruff (linting and formatting)\n#\n\n[tool.ruff]\n# Target Python 3.7+ (minimum version supported by ruff)\ntarget-version = "py37"\n\n# Exclude files and directories\nextend-exclude = [\n    "*_pb2.py",     # Protocol Buffer files (covers all pb2 files including grpc_test_service_pb2.py)\n    "*_pb2_grpc.py", # Protocol Buffer files (covers all pb2_grpc files including grpc_test_service_pb2_grpc.py)\n    "checkouts",    # From flake8\n    "lol*",         # From flake8\n]\n\n[tool.ruff.lint]\n# Match flake8\'s default rule selection exactly\n# Flake8 by default only enables E and W (pycodestyle) + F (pyflakes)\nselect = [\n    "E",   # pycodestyle errors (same as flake8 default)\n    "W",   # pycodestyle warnings (same as flake8 default)\n    "F",   # Pyflakes (same as flake8 default)\n    # Note: B and N rules are NOT enabled by default in flake8\n    # They were only active through the plugins, which may not have been fully enabled\n]\n\n# Use ONLY the same ignores as the original flake8 config + compatibility for this codebase\nignore = [\n    "E203", # Whitespace before \':\'\n    "E501", # Line too long\n    "E402", # Module level import not at top of file\n    "E731", # Do not assign a lambda expression, use a def\n    "B014", # Redundant exception types\n    "N812", # Lowercase imported as non-lowercase\n    "N804", # First argument of classmethod should be named cls\n\n    # Additional ignores for codebase compatibility\n    "F401", # Unused imports - many in TYPE_CHECKING blocks used for type comments\n    "E721", # Use isinstance instead of type() == - existing pattern in this codebase\n]\n\n[tool.ruff.format]\n# ruff format already excludes the same files as specified in extend-exclude\n# Ensure Python 3.7 compatibility - avoid using Python 3.9+ syntax features\nskip-magic-trailing-comma = false\n'
                      data_file: /home/runner/work/sentry-python/sentry-python/.coverage
                         python: 3.14.0 (main, Oct  7 2025, 13:06:58) [GCC 11.4.0]
                       platform: Linux-6.8.0-1041-azure-x86_64-with-glibc2.35
                 implementation: CPython
                          build: main
                                 Oct  7 2025 13:06:58
                    gil_enabled: True
                     executable: /opt/hostedtoolcache/Python/3.14.0/x64/bin/python
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 24710
                            cwd: /home/runner/work/sentry-python/sentry-python
                           path: 
                                 /opt/hostedtoolcache/Python/3.14.0/x64/bin
                                 /opt/hostedtoolcache/Python/3.14.0/x64/lib/python314.zip
                                 /opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14
                                 /opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/lib-dynload
                                 /opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/site-packages
                    environment: HOME = /home/runner
                   command_line: /opt/hostedtoolcache/Python/3.14.0/x64/bin/coverage xml --debug=sys,core
         sqlite3_sqlite_version: 3.37.2
             sqlite3_temp_store: 0
        sqlite3_compile_options: ATOMIC_INTRINSICS=1, COMPILER=gcc-11.4.0, DEFAULT_AUTOVACUUM,
                                 DEFAULT_CACHE_SIZE=-2000, DEFAULT_FILE_FORMAT=4,
                                 DEFAULT_JOURNAL_SIZE_LIMIT=-1, DEFAULT_MMAP_SIZE=0, DEFAULT_PAGE_SIZE=4096,
                                 DEFAULT_PCACHE_INITSZ=20, DEFAULT_RECURSIVE_TRIGGERS,
                                 DEFAULT_SECTOR_SIZE=4096, DEFAULT_SYNCHRONOUS=2,
                                 DEFAULT_WAL_AUTOCHECKPOINT=1000, DEFAULT_WAL_SYNCHRONOUS=2,
                                 DEFAULT_WORKER_THREADS=0, ENABLE_COLUMN_METADATA, ENABLE_DBSTAT_VTAB,
                                 ENABLE_FTS3, ENABLE_FTS3_PARENTHESIS, ENABLE_FTS3_TOKENIZER, ENABLE_FTS4,
                                 ENABLE_FTS5, ENABLE_JSON1, ENABLE_LOAD_EXTENSION, ENABLE_MATH_FUNCTIONS,
                                 ENABLE_PREUPDATE_HOOK, ENABLE_RTREE, ENABLE_SESSION, ENABLE_STMTVTAB,
                                 ENABLE_UNLOCK_NOTIFY, ENABLE_UPDATE_DELETE_LIMIT, HAVE_ISNAN,
                                 LIKE_DOESNT_MATCH_BLOBS, MALLOC_SOFT_LIMIT=1024, MAX_ATTACHED=10,
                                 MAX_COLUMN=2000, MAX_COMPOUND_SELECT=500, MAX_DEFAULT_PAGE_SIZE=32768,
                                 MAX_EXPR_DEPTH=1000, MAX_FUNCTION_ARG=127, MAX_LENGTH=1000000000,
                                 MAX_LIKE_PATTERN_LENGTH=50000, MAX_MMAP_SIZE=0x7fff0000,
                                 MAX_PAGE_COUNT=1073741823, MAX_PAGE_SIZE=65536, MAX_SCHEMA_RETRY=25,
                                 MAX_SQL_LENGTH=1000000000, MAX_TRIGGER_DEPTH=1000,
                                 MAX_VARIABLE_NUMBER=250000, MAX_VDBE_OP=250000000, MAX_WORKER_THREADS=8,
                                 MUTEX_PTHREADS, OMIT_LOOKASIDE, SECURE_DELETE, SOUNDEX, SYSTEM_MALLOC,
                                 TEMP_STORE=1, THREADSAFE=1, USE_URI
-- end -------------------------------------------------------

sentrivana avatar Nov 13 '25 08:11 sentrivana

This issue might be related and might help to find a common cause: https://github.com/getsentry/sentry-python/pull/5088

Could it be a conflict between sysmon and apps using multiprocessing through the forkserver as the new default context on Linux systems?

MRigal avatar Nov 13 '25 09:11 MRigal

Just adding some observations (also from the above sentry-python repo).

We also use pytest-forked for some of our tests. Could it be that the removal of this optimization along with our forked tests causes a lot of slowdown?

sl0thentr0py avatar Nov 13 '25 12:11 sl0thentr0py

Thanks for the clues. Also, for the getsentry reproduction. I can confirm this is down to coverage.py and the Python version somehow:

% tox -qe py3.11-common -- -k test_basic
.........................................................................ss...............sss............................................................. [ 87%]
......................                                                                                                                                     [100%]
  py3.11-common: OK (12.58 seconds)
  congratulations :) (14.35 seconds)

% tox -qe py3.14-common -- -k test_basic
.........................................................................ss...............sss............................................................. [ 87%]
......................                                                                                                                                     [100%]
  py3.14-common: OK (40.41 seconds)
  congratulations :) (42.19 seconds)

nedbat avatar Nov 13 '25 12:11 nedbat

The bulk of the problem is due to commit 31f91f816244a89051b301f2e3536e5f7b951e3d. Resetting coverage to just before that commit, I get: 19.90s instead of 40.41s.

But this is still much slower than 3.11 which is 12.58s. If I force 3.14 to use the "ctrace" core instead, then I get 14.13s, which is better, but still slower than 3.11.

So something unusual is going on, and it seems like whatever it is, getsentry does it more than other repos.

nedbat avatar Nov 13 '25 13:11 nedbat

Maybe it has to do with the small size of the cache in https://github.com/coveragepy/coveragepy/commit/31f91f816244a89051b301f2e3536e5f7b951e3d#diff-3b2682bb62c0b1d94ddeb64f19bf754f87c096aa3c695b812eb0e976ed2303d2R469

Which may affect negatively particularly big repos?

MRigal avatar Nov 13 '25 13:11 MRigal

@nedbat our test suite is indeed a bit of a monstrosity of monkeypatching and forking, so 14.13s instead of 12.58s seems reasonable enough to me. So summarizing a bit - we could use ctrace also on 3.14 and for some reason, sysmon is particularly bad on our repo.

edit: can verify that forcing ctrace is back to ~4m runs

sl0thentr0py avatar Nov 13 '25 14:11 sl0thentr0py

I think this has to do with a proliferation of code objects. The basis of sysmon is that it can disable firing an event after it's been fired once, reducing the overhead of measuring. It tracks that information on (or keyed by) code objects. When I run -k test_basics, there are about 120 tests, and 88 distinct code objects all from sentry_sdk/integrations/starlette.py (to pick just one file). This is all within a single process, so it's not about pytest-forked. The same file is being compiled somehow in the same process.

This defeats the sysmon premise (events are getting fired 88x times more in that file, and I'm sure others), and some of coverage.py's avoidance of work as well.

I don't know what we can do about this in either CPython or coverage.py.

nedbat avatar Nov 13 '25 15:11 nedbat

I measured the unique code objects incorrectly, but it's still a lot. Looks like many of the /sentry_sdk/integrations/*.py files have 40 distinct code objects.

nedbat avatar Nov 14 '25 12:11 nedbat

I've been experimenting with the getsentry code in a fork where I can hack away at stuff: https://github.com/nedbat/sentry-python/tree/nedbat/debugging-2082

nedbat avatar Nov 15 '25 14:11 nedbat

Coming from https://github.com/pytest-dev/pytest/pull/13991, per request from @nedbat, here is the output of running pytest with sysmon under Python 3.14, COVERAGE_SYSMON_LOG=1, commit d50201bc2306a31d16a478e1ed182d835d4d4602. Note the file has 3M lines and decompresses to 417MB.

foo.zip

bluetech avatar Nov 22 '25 19:11 bluetech

FWIW I have a relatively small project that is affected by this: https://codeberg.org/jepler/wwvbpy/ at 97bb1601e8ea9e246df698547f8c845156ddb1e6, the current tip of main.

My code uses branch coverage & this test is specifically a test of subprocesses.

# pyproject.toml fragment
[tool.coverage.run]
patch=["subprocess"]
branch=true

I ran these tests in a docker container of astral uv on my linux laptop (intel i5-1235U with plenty RAM & debian trixie):

wwvbpy$ docker run --rm -it -v=.:/work -w /work ghcr.io/astral-sh/uv:debian bash
# for python in 3.13 3.14 ; do for coverage in 7.11.0 7.11.1; do echo "PYTHON $python with coveragepy $coverage"; uv run --managed-python --with-requirements requirements-dev.txt --python $python --with coverage==$coverage python -m coverage run -m unittest test/testcli.py 2>&1 | grep 'Ran . tests in'; done; done
PYTHON 3.13 with coveragepy 7.11.0
Ran 6 tests in 1.511s
PYTHON 3.13 with coveragepy 7.11.1
Ran 6 tests in 1.475s
PYTHON 3.14 with coveragepy 7.11.0
Ran 6 tests in 2.951s
PYTHON 3.14 with coveragepy 7.11.1
Ran 6 tests in 7.076s

For me, 3.14/7.11.0 took about 2x as long as 3.13/7.11.0, and then the switch from 7.11.0 to 7.11.1 more than doubled things again. The total performance regression from the best combination (3.13/7.11.0) to the worst (3.14/7.11.1) was about 4.6x runtime.

jepler avatar Dec 05 '25 19:12 jepler

coverage debug config for various versions

PYTHON 3.13 with coveragepy 7.11.0
-- config ----------------------------------------------------
                         branch: True
                   command_line: None
                    concurrency: -none-
                    config_file: /work/pyproject.toml
         config_files_attempted: /work/.coveragerc
                                 /work/setup.cfg
                                 /work/tox.ini
                                 /work/pyproject.toml
              config_files_read: /work/pyproject.toml
                        context: None
                           core: None
                    cover_pylib: False
                      data_file: .coverage
                          debug: -none-
                     debug_file: None
               disable_warnings: -none-
                dynamic_context: None
                   exclude_also: -none-
                   exclude_list: #\s*(pragma|PRAGMA)[:\s]?\s*(no|NO)\s*(cover|COVER)
                                 ^\s*(((async )?def .*?)?\)(\s*->.*?)?:\s*)?\.\.\.\s*(#|$)
                                 if (typing\.)?TYPE_CHECKING:
                      extra_css: None
                     fail_under: 0.0
                         format: None
                       html_dir: htmlcov
              html_skip_covered: None
                html_skip_empty: None
                     html_title: Coverage report
                  ignore_errors: False
     include_namespace_packages: False
                    json_output: coverage.json
              json_pretty_print: False
             json_show_contexts: False
            lcov_line_checksums: False
                    lcov_output: coverage.lcov
                       parallel: True
                   partial_also: -none-
            partial_always_list: while (True|1|False|0):
                                 if (True|1|False|0):
                   partial_list: #\s*(pragma|PRAGMA)[:\s]?\s*(no|NO)\s*(branch|BRANCH)
                          patch: subprocess
                          paths: {}
                 plugin_options: {}
                        plugins: -none-
                      precision: 0
                 relative_files: False
                report_contexts: None
                 report_include: None
                    report_omit: None
                    run_include: -none-
                       run_omit: -none-
                  show_contexts: False
                   show_missing: False
                        sigterm: False
                   skip_covered: False
                     skip_empty: False
                           sort: None
                         source: None
                    source_dirs: -none-
                    source_pkgs: -none-
                          timid: False
                     xml_output: coverage.xml
              xml_package_depth: 99
PYTHON 3.13 with coveragepy 7.11.1
-- config ----------------------------------------------------
                         branch: True
                   command_line: None
                    concurrency: -none-
                    config_file: /work/pyproject.toml
         config_files_attempted: /work/.coveragerc
                                 /work/setup.cfg
                                 /work/tox.ini
                                 /work/pyproject.toml
              config_files_read: /work/pyproject.toml
                        context: None
                           core: None
                    cover_pylib: False
                      data_file: .coverage
                          debug: -none-
                     debug_file: None
               disable_warnings: -none-
                dynamic_context: None
                   exclude_also: -none-
                   exclude_list: #\s*(pragma|PRAGMA)[:\s]?\s*(no|NO)\s*(cover|COVER)
                                 ^\s*(((async )?def .*?)?\)(\s*->.*?)?:\s*)?\.\.\.\s*(#|$)
                                 if (typing\.)?TYPE_CHECKING:
                      extra_css: None
                     fail_under: 0.0
                         format: None
                       html_dir: htmlcov
              html_skip_covered: None
                html_skip_empty: None
                     html_title: Coverage report
                  ignore_errors: False
     include_namespace_packages: False
                    json_output: coverage.json
              json_pretty_print: False
             json_show_contexts: False
            lcov_line_checksums: False
                    lcov_output: coverage.lcov
                       parallel: True
                   partial_also: -none-
            partial_always_list: while (True|1|False|0):
                                 if (True|1|False|0):
                   partial_list: #\s*(pragma|PRAGMA)[:\s]?\s*(no|NO)\s*(branch|BRANCH)
                          patch: subprocess
                          paths: {}
                 plugin_options: {}
                        plugins: -none-
                      precision: 0
                 relative_files: False
                report_contexts: None
                 report_include: None
                    report_omit: None
                    run_include: -none-
                       run_omit: -none-
                  show_contexts: False
                   show_missing: False
                        sigterm: False
                   skip_covered: False
                     skip_empty: False
                           sort: None
                         source: None
                    source_dirs: -none-
                    source_pkgs: -none-
                          timid: False
                     xml_output: coverage.xml
              xml_package_depth: 99
PYTHON 3.14 with coveragepy 7.11.0
-- config ----------------------------------------------------
                         branch: True
                   command_line: None
                    concurrency: -none-
                    config_file: /work/pyproject.toml
         config_files_attempted: /work/.coveragerc
                                 /work/setup.cfg
                                 /work/tox.ini
                                 /work/pyproject.toml
              config_files_read: /work/pyproject.toml
                        context: None
                           core: None
                    cover_pylib: False
                      data_file: .coverage
                          debug: -none-
                     debug_file: None
               disable_warnings: -none-
                dynamic_context: None
                   exclude_also: -none-
                   exclude_list: #\s*(pragma|PRAGMA)[:\s]?\s*(no|NO)\s*(cover|COVER)
                                 ^\s*(((async )?def .*?)?\)(\s*->.*?)?:\s*)?\.\.\.\s*(#|$)
                                 if (typing\.)?TYPE_CHECKING:
                      extra_css: None
                     fail_under: 0.0
                         format: None
                       html_dir: htmlcov
              html_skip_covered: None
                html_skip_empty: None
                     html_title: Coverage report
                  ignore_errors: False
     include_namespace_packages: False
                    json_output: coverage.json
              json_pretty_print: False
             json_show_contexts: False
            lcov_line_checksums: False
                    lcov_output: coverage.lcov
                       parallel: True
                   partial_also: -none-
            partial_always_list: while (True|1|False|0):
                                 if (True|1|False|0):
                   partial_list: #\s*(pragma|PRAGMA)[:\s]?\s*(no|NO)\s*(branch|BRANCH)
                          patch: subprocess
                          paths: {}
                 plugin_options: {}
                        plugins: -none-
                      precision: 0
                 relative_files: False
                report_contexts: None
                 report_include: None
                    report_omit: None
                    run_include: -none-
                       run_omit: -none-
                  show_contexts: False
                   show_missing: False
                        sigterm: False
                   skip_covered: False
                     skip_empty: False
                           sort: None
                         source: None
                    source_dirs: -none-
                    source_pkgs: -none-
                          timid: False
                     xml_output: coverage.xml
              xml_package_depth: 99
PYTHON 3.14 with coveragepy 7.11.1
-- config ----------------------------------------------------
                         branch: True
                   command_line: None
                    concurrency: -none-
                    config_file: /work/pyproject.toml
         config_files_attempted: /work/.coveragerc
                                 /work/setup.cfg
                                 /work/tox.ini
                                 /work/pyproject.toml
              config_files_read: /work/pyproject.toml
                        context: None
                           core: None
                    cover_pylib: False
                      data_file: .coverage
                          debug: -none-
                     debug_file: None
               disable_warnings: -none-
                dynamic_context: None
                   exclude_also: -none-
                   exclude_list: #\s*(pragma|PRAGMA)[:\s]?\s*(no|NO)\s*(cover|COVER)
                                 ^\s*(((async )?def .*?)?\)(\s*->.*?)?:\s*)?\.\.\.\s*(#|$)
                                 if (typing\.)?TYPE_CHECKING:
                      extra_css: None
                     fail_under: 0.0
                         format: None
                       html_dir: htmlcov
              html_skip_covered: None
                html_skip_empty: None
                     html_title: Coverage report
                  ignore_errors: False
     include_namespace_packages: False
                    json_output: coverage.json
              json_pretty_print: False
             json_show_contexts: False
            lcov_line_checksums: False
                    lcov_output: coverage.lcov
                       parallel: True
                   partial_also: -none-
            partial_always_list: while (True|1|False|0):
                                 if (True|1|False|0):
                   partial_list: #\s*(pragma|PRAGMA)[:\s]?\s*(no|NO)\s*(branch|BRANCH)
                          patch: subprocess
                          paths: {}
                 plugin_options: {}
                        plugins: -none-
                      precision: 0
                 relative_files: False
                report_contexts: None
                 report_include: None
                    report_omit: None
                    run_include: -none-
                       run_omit: -none-
                  show_contexts: False
                   show_missing: False
                        sigterm: False
                   skip_covered: False
                     skip_empty: False
                           sort: None
                         source: None
                    source_dirs: -none-
                    source_pkgs: -none-
                          timid: False
                     xml_output: coverage.xml
              xml_package_depth: 99

jepler avatar Dec 05 '25 20:12 jepler

If I turn off either the subprocess patch or branch coverage the speeds are comparable across python & coverage versions. The specific test file I selected is the one that makes heavy use of subprocess calls, about a dozen or so.

If I force the tracer core to ctrace, time is comparable across versions:

# for python in 3.13 3.14 ; do for coverage in 7.11.0 7.11.1 7.11.3; do echo "PYTHON $python with coveragepy $coverage"; uv run --managed-python --with-requirements requirements-dev.txt --python $python --with coverage==$coverage python -m coverage  run --debug=sys,core -m unittest test/testcli.py 2>&1 | grep 'Ran.* tests in ' ; done; done
PYTHON 3.13 with coveragepy 7.11.0
Ran 6 tests in 1.781s
PYTHON 3.13 with coveragepy 7.11.1
Ran 6 tests in 1.539s
PYTHON 3.13 with coveragepy 7.11.3
Ran 6 tests in 1.552s
PYTHON 3.14 with coveragepy 7.11.0
Ran 6 tests in 1.648s
PYTHON 3.14 with coveragepy 7.11.1
Ran 6 tests in 1.571s
PYTHON 3.14 with coveragepy 7.11.3
Ran 6 tests in 1.684s
 [tool.coverage.run]
 patch=["subprocess"]
 branch=true
+core="ctrace"

jepler avatar Dec 06 '25 01:12 jepler

I haven't had a chance to try this yet, but it surprises me that turning off the subprocess patch changes the timings. Do you mind trying the latest main branch of this repo to see how it behaves?

nedbat avatar Dec 06 '25 02:12 nedbat

Doesn't turning off subprocess patching mean the sub-programs aren't actually run under the coverage tracer? Subprocesses are the majority of the runtime in my reproducer, so it's not a surprise that changing this setting makes a difference.

jepler avatar Dec 06 '25 02:12 jepler

pyproject settings back to original committed version. It does seem a slight improvement compared to 7.11.3.

root@9b12b98dacd8:/work# for python in 3.10 3.13 3.14 ; do for coverage in coverage==7.11.0 coverage==7.11.3 git+https://github.com/coveragepy/coveragepy@d5e7c3ad0d5; do echo "PYTHON $python with coveragepy $coverage"; uv run --managed-python --with-requirements requirements-dev.txt --python $python --with $coverage python -m coverage  run --debug=sys,core -m unittest test/testcli.py 2>&1 | grep 'Ran.* tests in ' ; done; done
PYTHON 3.10 with coveragepy coverage==7.11.0
Ran 6 tests in 1.401s
PYTHON 3.10 with coveragepy coverage==7.11.3
Ran 6 tests in 1.429s
PYTHON 3.10 with coveragepy git+https://github.com/coveragepy/coveragepy@d5e7c3ad0d5
Ran 6 tests in 1.390s
PYTHON 3.13 with coveragepy coverage==7.11.0
Ran 6 tests in 1.560s
PYTHON 3.13 with coveragepy coverage==7.11.3
Ran 6 tests in 1.545s
PYTHON 3.13 with coveragepy git+https://github.com/coveragepy/coveragepy@d5e7c3ad0d5
Ran 6 tests in 1.570s
PYTHON 3.14 with coveragepy coverage==7.11.0
Ran 6 tests in 2.992s
PYTHON 3.14 with coveragepy coverage==7.11.3
Ran 6 tests in 6.994s
PYTHON 3.14 with coveragepy git+https://github.com/coveragepy/coveragepy@d5e7c3ad0d5
Ran 6 tests in 5.434s

jepler avatar Dec 06 '25 02:12 jepler

@jepler Thanks for the reproduction instructions. I've done some experiments with your repo, and I have found an odd thing.

I cloned https://codeberg.org/jepler/wwvbpy/ at commit 97bb160 to reproduce your problem.

To use git bisect on my source, I had to add --reinstall --refresh --no-cache to uv run to get it to truly use the current source files. Lesson learned. Bisecting showed that https://github.com/nedbat/coveragepy/commit/31f91f816244a89051b301f2e3536e5f7b951e3d was the problem commit, which makes total sense if you read the commit messages between 7.11.0 and 7.11.1.

The truly odd thing is that uv run runs your tests slower than making a traditional venv and using pip! I don't understand why, but I want to dig into it more. The bad commit makes both environment styles slower, but the slowdown factor is worse for uv that for venv. So uv starts out slower, and the bad commit makes it slower worse than venv.

I adapted your experiment like this:

for coverage in coverage==7.11.0 git+https://github.com/coveragepy/coveragepy@31f91f816244a8 coverage==7.11.1; do
    for core in sysmon; do
        rm -rf .venv
        echo "venv+pip: coveragepy $coverage core $core"
        python3.14 -m venv .venv
        .venv/bin/python -m pip install -q $coverage 
        .venv/bin/python -m pip install -q .
        env PYTHONPATH=src COVERAGE_CORE=$core .venv/bin/python -m coverage run -m unittest discover -s test 2>&1 | grep 'Ran'
        rm -rf .venv
        echo "uv run: coveragepy $coverage core $core"
        env PYTHONPATH=src COVERAGE_CORE=$core uv run --python=3.14 --reinstall --refresh --no-cache --with $coverage python -m coverage run -m unittest discover -s test 2>&1 | grep 'Ran'
        rm -rf .venv
    done
done

The results:

venv+pip: coveragepy coverage==7.11.0 core sysmon
Ran 43 tests in 2.602s
uv run: coveragepy coverage==7.11.0 core sysmon
Ran 43 tests in 4.123s
venv+pip: coveragepy git+https://github.com/coveragepy/coveragepy@31f91f816244a8 core sysmon
Ran 43 tests in 3.007s
uv run: coveragepy git+https://github.com/coveragepy/coveragepy@31f91f816244a8 core sysmon
Ran 43 tests in 8.064s
venv+pip: coveragepy coverage==7.11.1 core sysmon
Ran 43 tests in 2.955s
uv run: coveragepy coverage==7.11.1 core sysmon
Ran 43 tests in 8.006s

In table form:

Coverage version venv+pip uv run slower
7.11.0 2.602s 4.123s 58%
7.11.1 2.955s 8.006s 70%
slower 13% 94%

The results were very reproducible over a number of runs.

nedbat avatar Dec 08 '25 15:12 nedbat

I reproduce the same: uv python 3.14 is markedly slower than classic venv at running my tests under coverage with the sysmon tracer (everything running in python:3.14-trixie container)

I considered that it could be startup time of my spawned processes but most of the difference is erased when using the ctrace core.

I thought it might be due to uv preferring 3.14t but using your script in the python:3.14-trixie container uses the same /usr/local/bin/python3.14 for uv & classic venv. An earlier script of mine used --managed-python, meaning the container python would never be used; apparently by default uv prefers 3.14t over the GIL version when it has to download python.

jepler avatar Dec 08 '25 18:12 jepler

I took the liberty of raising an issue with uv, in case it can bring useful eyes: https://github.com/astral-sh/uv/issues/17041

jepler avatar Dec 09 '25 02:12 jepler