Fails to build on Python 3.11 - longintrepr.h: No such file or directory

[gaborbernat]

Describe the bug

Because of https://github.com/python/cpython/pull/28968, cython fixed it in https://github.com/cython/cython/pull/4428 and is released with 0.29.5

To Reproduce

pip install aiohttp

Expected behavior

To build the wheel.

Logs/tracebacks

Failed to build aiohttp yarl
  error: subprocess-exited-with-error

  × Building wheel for aiohttp (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [31 lines of output]
      *********************
      * Accelerated build *
      *********************
      running bdist_wheel
      running build
      running build_py
      running egg_info
      writing aiohttp.egg-info/PKG-INFO
      writing dependency_links to aiohttp.egg-info/dependency_links.txt
      writing requirements to aiohttp.egg-info/requires.txt
      writing top-level names to aiohttp.egg-info/top_level.txt
      reading manifest file 'aiohttp.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching 'aiohttp' anywhere in distribution
      warning: no previously-included files matching '*.pyc' found anywhere in distribution
      warning: no previously-included files matching '*.pyd' found anywhere in distribution
      warning: no previously-included files matching '*.so' found anywhere in distribution
      warning: no previously-included files matching '*.lib' found anywhere in distribution
      warning: no previously-included files matching '*.dll' found anywhere in distribution
      warning: no previously-included files matching '*.a' found anywhere in distribution
      warning: no previously-included files matching '*.obj' found anywhere in distribution
      warning: no previously-included files found matching 'aiohttp/*.html'
      no previously-included directories found matching 'docs/_build'
      adding license file 'LICENSE.txt'
      running build_ext
      building 'aiohttp._websocket' extension
      aiohttp/_websocket.c:198:12: fatal error: longintrepr.h: No such file or directory
        198 |   #include "longintrepr.h"
            |            ^~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

Python Version

3.11

aiohttp Version

latest

multidict Version

n/a

yarl Version

latest

OS

Any

Related component

Server, Client

Additional context

No response

Code of Conduct

• [X] I agree to follow the aio-libs Code of Conduct

[webknjaz]

Thanks! It usually doesn't make sense for us to be working on the next interpreter support until rc are out because the build deps like Cython can't keep up with the ABI changes anyway.

[gaborbernat]

Perhaps RC is a bit too late, IMHO beta should be ok, no? 😊 (this blocks us from testing tox)

[webknjaz]

I don't think it's up to us really. We can enable testing in the CI (I would be actually in favor if that was realistic) but Cython having to go through upgrades first is traditionally a blocker that is out of our control. This usually results in this effort being rather pointless until the ABI is stable and Cython supports the respective changes. Also, Cython still isn't able to produce abi3 wheels which I suspect would greatly improve the situation for everyone.

[webknjaz]

@gaborbernat you say that it's fixed in Cython that is released already but the problem is building wheels from the published sdist? Are you asking for the release of a new version with C-files generated with newer Cython? If that's an action item right now, I think that it's doable.

[gaborbernat]

Are you asking for the release of a new version with C-files generated with newer Cython? If that's an action item right now, I think that it's doable.

I think this is the case 🤔

[sagar1025]

Are you asking for the release of a new version with C-files generated with newer Cython? If that's an action item right now, I think that it's doable.

I think this is the case 🤔

Seems that way. https://github.com/cython/cython/pull/4428

[droserasprout]

Python 3.11.0b4 is out. Should we revisit this one? Citing release notes here:

Our goal is have no ABI changes after beta 4 and as few code changes as possible after 3.11.0rc1, the first release candidate. To achieve that, it will be extremely important to get as much exposure for 3.11 as possible during the beta phase.

[cclauss]

• #6830
• https://github.com/aio-libs/aiohttp/pull/6830/files

[gaborbernat]

The main branch is already fixed, someone just need to cut a release I believe :thinking:

[Dreamsorcerer]

Not that I'm aware of. Tests are still failing for 3.11.

[webknjaz]

The main branch is already fixed, someone just need to cut a release I believe thinking

Note that the master branch corresponds to aiohttp 4 which does not have stable releases. We release production-quality versions from the stable branches.

[cclauss]

Several key dependencies (e.g. cython, frozenlist, and yarl) now have new Py3.11 releases.

[webknjaz]

So with updated deps (frozenlist, yarl, cython and pytest), the CI fails but the output isn't useful. I reproduced it locally and it left me confused — the pytest invocation produces zero output no matter how much verbosity I request from it and the return code is 1:

$ pytest
$ echo $?
1
$ pytest -vvvvv
1
$ python -m pytest
$ echo $?
1
$ python -Xdev -m pytest
$ echo $?
1
$ python -X dev -X faulthandler -X importtime -m pytest
import time: self [us] | cumulative | imported package
import time:       139 |        139 |   _io
import time:        30 |         30 |   marshal
import time:       306 |        306 |   posix
import time:       282 |        756 | _frozen_importlib_external
import time:        81 |         81 |   time
import time:        92 |        172 | zipimport
import time:        90 |         90 | faulthandler
import time:        87 |         87 |     _codecs
import time:       175 |        261 |   codecs
import time:       315 |        315 |   encodings.aliases
import time:       396 |        971 | encodings
import time:       181 |        181 | encodings.utf_8
import time:        72 |         72 | _signal
import time:        27 |         27 |     _abc
import time:       102 |        128 |   abc
import time:       136 |        263 | io
import time:       260 |        260 | warnings
import time:        37 |         37 |       _stat
import time:        49 |         86 |     stat
import time:       728 |        728 |     _collections_abc
import time:        28 |         28 |       genericpath
import time:        53 |         81 |     posixpath
import time:       271 |       1164 |   os
import time:        58 |         58 |   _sitebuiltins
import time:       235 |        235 |   types
import time:       144 |        144 |     importlib
import time:       154 |        154 |     importlib._abc
import time:        85 |         85 |         itertools
import time:       108 |        108 |         keyword
import time:        61 |         61 |           _operator
import time:       241 |        302 |         operator
import time:       149 |        149 |         reprlib
import time:        52 |         52 |         _collections
import time:       808 |       1501 |       collections
import time:        52 |         52 |         _functools
import time:       456 |        507 |       functools
import time:       600 |       2607 |     contextlib
import time:       115 |       3018 |   importlib.util
import time:        54 |         54 |   importlib.machinery
import time:       282 |        282 |   sitecustomize
import time:      5248 |      10055 | site
import time:       130 |        130 | runpy
import time:       147 |        147 |     _pytest._version
import time:       228 |        375 |   _pytest
import time:      1094 |       1094 |         _ast
import time:      1364 |       1364 |         enum
import time:      1812 |       4269 |       ast
import time:       193 |        193 |               encodings.ascii
import time:       153 |        346 |             _opcode
import time:       331 |        676 |           opcode
import time:       802 |       1477 |         dis
import time:       153 |        153 |         collections.abc
import time:        62 |         62 |                 _sre
import time:       295 |        295 |                   re._constants
import time:       294 |        588 |                 re._parser
import time:       110 |        110 |                 re._casefix
import time:       247 |       1006 |               re._compiler
import time:       145 |        145 |               copyreg
import time:       424 |       1574 |             re
import time:       152 |        152 |             token
import time:       196 |        196 |             encodings.latin_1
import time:      1020 |       2942 |           tokenize
import time:       132 |       3073 |         linecache
import time:      1747 |       6449 |       inspect
import time:      1057 |       1057 |         textwrap
import time:       662 |       1718 |       traceback
import time:       152 |        152 |         fnmatch
import time:        70 |         70 |           _winapi
import time:        59 |         59 |           nt
import time:        54 |         54 |           nt
import time:        51 |         51 |           nt
import time:        51 |         51 |           nt
import time:        51 |         51 |           nt
import time:        86 |        419 |         ntpath
import time:        53 |         53 |         errno
import time:       107 |        107 |           urllib
import time:      1006 |       1112 |         urllib.parse
import time:       837 |       2572 |       pathlib
import time:       118 |        118 |         _typing
import time:      2127 |       2244 |       typing
import time:       284 |        284 |         _weakrefset
import time:       401 |        684 |       weakref
import time:       134 |        134 |         __future__
import time:      3054 |       3054 |             platform
import time:       205 |       3259 |           attr._compat
import time:        67 |         67 |                   org
import time:        14 |         80 |                 org.python
import time:        14 |         94 |               org.python.core
import time:       188 |        282 |             copy
import time:       558 |        558 |             threading
import time:       183 |        183 |               _uuid
import time:       374 |        556 |             uuid
import time:       102 |        102 |             attr._config
import time:       243 |        243 |               attr.exceptions
import time:       153 |        396 |             attr.setters
import time:      1946 |       3838 |           attr._make
import time:       176 |       7271 |         attr.converters
import time:       117 |        117 |         attr.filters
import time:      3876 |       3876 |         attr.validators
import time:       178 |        178 |         attr._cmp
import time:       147 |        147 |         attr._funcs
import time:       500 |        500 |         attr._version_info
import time:       143 |        143 |         attr._next_gen
import time:       279 |      12641 |       attr
import time:       124 |        124 |         pluggy._version
import time:       138 |        138 |           pluggy._tracing
import time:       131 |        131 |             pluggy._result
import time:       112 |        242 |           pluggy._callers
import time:       220 |        220 |           pluggy._hooks
import time:       196 |        196 |               _csv
import time:       371 |        566 |             csv
import time:       145 |        145 |             email
import time:       212 |        212 |               binascii
import time:       150 |        150 |                 zlib
import time:       185 |        185 |                   _compression
import time:       167 |        167 |                   _bz2
import time:       204 |        555 |                 bz2
import time:       206 |        206 |                   _lzma
import time:       266 |        471 |                 lzma
import time:       537 |       1712 |               shutil
import time:       136 |        136 |                 _struct
import time:       115 |        251 |               struct
import time:       978 |       3151 |             zipfile
import time:       139 |        139 |                 quopri
import time:       142 |        142 |                     math
import time:        99 |         99 |                       _bisect
import time:       121 |        219 |                     bisect
import time:       101 |        101 |                     _random
import time:       101 |        101 |                     _sha512
import time:       321 |        882 |                   random
import time:       255 |        255 |                     _socket
import time:       130 |        130 |                       select
import time:       463 |        592 |                     selectors
import time:       174 |        174 |                     array
import time:      1485 |       2504 |                   socket
import time:       183 |        183 |                     _datetime
import time:       780 |        962 |                   datetime
import time:        67 |         67 |                         _locale
import time:       807 |        873 |                       locale
import time:       408 |       1281 |                     calendar
import time:       187 |       1467 |                   email._parseaddr
import time:       196 |        196 |                       base64
import time:       110 |        306 |                     email.base64mime
import time:        27 |         27 |                         _string
import time:       551 |        578 |                       string
import time:       235 |        812 |                     email.quoprimime
import time:       435 |        435 |                     email.errors
import time:       118 |        118 |                     email.encoders
import time:       186 |       1855 |                   email.charset
import time:       387 |       8055 |                 email.utils
import time:       600 |        600 |                   email.header
import time:      1108 |       1708 |                 email._policybase
import time:       240 |        240 |                 email._encoded_words
import time:       117 |        117 |                 email.iterators
import time:       553 |      10809 |               email.message
import time:        96 |         96 |                 importlib.metadata._functools
import time:       150 |        246 |               importlib.metadata._text
import time:       233 |      11287 |             importlib.metadata._adapters
import time:       306 |        306 |             importlib.metadata._meta
import time:       252 |        252 |             importlib.metadata._collections
import time:       101 |        101 |             importlib.metadata._itertools
import time:       926 |        926 |                     tempfile
import time:       311 |        311 |                     importlib.resources.abc
import time:       262 |        262 |                     importlib.resources._adapters
import time:       248 |       1745 |                   importlib.resources._common
import time:       170 |        170 |                   importlib.resources._legacy
import time:       123 |       2038 |                 importlib.resources
import time:        16 |       2054 |               importlib.resources.abc
import time:       371 |       2424 |             importlib.abc
import time:      1263 |      19493 |           importlib.metadata
import time:       329 |      20422 |         pluggy._manager
import time:       162 |      20707 |       pluggy
import time:       368 |        368 |       _pytest._code.source
import time:       157 |        157 |             unicodedata
import time:       105 |        262 |           _pytest._io.wcwidth
import time:       187 |        187 |               py._error
import time:        99 |         99 |               py._vendored_packages
import time:       113 |        113 |                 py._vendored_packages.apipkg.version
import time:       230 |        342 |               py._vendored_packages.apipkg
import time:        85 |         85 |               py._version
import time:       351 |       1062 |             py
import time:        99 |         99 |               py._path
import time:        30 |         30 |               atexit
import time:       312 |        312 |               py._path.common
import time:       442 |        881 |             py._path.local
import time:       887 |       2829 |           _pytest.compat
import time:       257 |       3347 |         _pytest._io.terminalwriter
import time:       117 |       3463 |       _pytest._io
import time:       583 |        583 |           dataclasses
import time:       256 |        839 |         pprint
import time:       193 |       1031 |       _pytest._io.saferepr
import time:       716 |        716 |         _pytest.warning_types
import time:       126 |        842 |       _pytest.deprecated
import time:       432 |        432 |         _pytest.outcomes
import time:       599 |       1031 |       _pytest.pathlib
import time:      5146 |      63159 |     _pytest._code.code
import time:       146 |      63304 |   _pytest._code
import time:       669 |        669 |             gettext
import time:       844 |       1513 |           argparse
import time:       351 |        351 |           shlex
import time:       987 |        987 |           _pytest.hookspec
import time:       163 |        163 |           _pytest.config.exceptions
import time:       211 |        211 |             iniconfig
import time:       265 |        476 |           _pytest.config.findpaths
import time:       216 |        216 |           _pytest.stash
import time:      1987 |       5690 |         _pytest.config
import time:       340 |       6030 |       _pytest.assertion.util
import time:      1010 |       1010 |               _pytest.mark.expression
import time:      1953 |       1953 |               _pytest.mark.structures
import time:       556 |        556 |               _pytest.config.argparsing
import time:      1743 |       5261 |             _pytest.mark
import time:        18 |       5278 |           _pytest.mark.structures
import time:       769 |       6047 |         _pytest.nodes
import time:       459 |        459 |           _pytest.scope
import time:      3691 |       4150 |         _pytest.fixtures
import time:       660 |        660 |         _pytest.reports
import time:       419 |        419 |           bdb
import time:       100 |        100 |           _pytest.timing
import time:      1049 |       1567 |         _pytest.runner
import time:      1123 |      13545 |       _pytest.main
import time:       768 |      20342 |     _pytest.assertion.rewrite
import time:       136 |        136 |     _pytest.assertion.truncate
import time:       296 |      20773 |   _pytest.assertion
import time:       149 |        149 |           _json
import time:       316 |        465 |         json.scanner
import time:       356 |        820 |       json.decoder
import time:       354 |        354 |       json.encoder
import time:       174 |       1347 |     json
import time:      2901 |       2901 |     _pytest.python
import time:       851 |       5098 |   _pytest.cacheprovider
import time:       851 |        851 |   _pytest.capture
import time:       404 |        404 |   _pytest.debugging
import time:       129 |        129 |   _pytest.freeze_support
import time:       529 |        529 |       signal
import time:       250 |        250 |       fcntl
import time:        66 |         66 |       msvcrt
import time:       100 |        100 |       _posixsubprocess
import time:       572 |       1515 |     subprocess
import time:       372 |        372 |     _pytest.monkeypatch
import time:        43 |         43 |       gc
import time:       657 |        657 |       _pytest.tmpdir
import time:      1237 |       1935 |     _pytest.pytester
import time:      1285 |       1285 |     _pytest.terminal
import time:       845 |       5950 |   _pytest.legacypath
import time:      2559 |       2559 |     logging
import time:      1214 |       3773 |   _pytest.logging
import time:       390 |        390 |         numbers
import time:       593 |        982 |       _decimal
import time:       126 |       1108 |     decimal
import time:       753 |       1860 |   _pytest.python_api
import time:       520 |        520 |   _pytest.recwarn
import time:       433 |     103465 | pytest
import time:       158 |        158 | _pytest.config.compat
import time:       222 |        222 | _pytest.helpconfig
import time:       494 |        494 | _pytest.unittest
import time:      1289 |       1289 | _pytest.skipping
import time:       173 |        173 | _pytest.pastebin
import time:       110 |        110 | _pytest.nose
import time:       115 |        115 |       xml
import time:       124 |        238 |     xml.etree
import time:       411 |        411 |     xml.etree.ElementPath
import time:       304 |        304 |       pyexpat
import time:       248 |        552 |     _elementtree
import time:       738 |       1938 |   xml.etree.ElementTree
import time:       607 |       2544 | _pytest.junitxml
import time:      1255 |       1255 | _pytest.doctest
import time:       239 |        239 | _pytest.setuponly
import time:       195 |        195 | _pytest.setupplan
import time:       235 |        235 | _pytest.stepwise
import time:       341 |        341 | _pytest.warnings
import time:       120 |        120 | _pytest.python_path
import time:       161 |        161 | _pytest.unraisableexception
import time:       158 |        158 | _pytest.threadexception
import time:       182 |        182 | _pytest.faulthandler
import time:       305 |        305 |   glob
import time:       157 |        462 | _pytest._argcomplete
import time:       116 |        116 |       tomli._types
import time:      1125 |       1241 |     tomli._re
import time:       634 |       1875 |   tomli._parser
import time:       169 |       2043 | tomli
import time:       124 |        124 |     packaging.__about__
import time:       169 |        292 |   packaging
import time:       132 |        132 |   packaging._structures
import time:      2116 |       2540 | packaging.version
import time:       127 |        127 |         concurrent
import time:       570 |        570 |         concurrent.futures._base
import time:       179 |        875 |       concurrent.futures
import time:       161 |        161 |         _heapq
import time:       188 |        348 |       heapq
import time:      1102 |       1102 |         _ssl
import time:      2141 |       3242 |       ssl
import time:       252 |        252 |       asyncio.constants
import time:       147 |        147 |       asyncio.coroutines
import time:       141 |        141 |           _contextvars
import time:       133 |        273 |         contextvars
import time:       143 |        143 |         asyncio.format_helpers
import time:       146 |        146 |           asyncio.base_futures
import time:       199 |        199 |           asyncio.exceptions
import time:       156 |        156 |           asyncio.base_tasks
import time:       237 |        736 |         _asyncio
import time:       525 |       1676 |       asyncio.events
import time:       329 |        329 |       asyncio.futures
import time:       202 |        202 |       asyncio.protocols
import time:       269 |        269 |         asyncio.transports
import time:       131 |        131 |         asyncio.log
import time:       757 |       1156 |       asyncio.sslproto
import time:       138 |        138 |           asyncio.mixins
import time:       400 |        400 |           asyncio.tasks
import time:       451 |        989 |         asyncio.locks
import time:       350 |       1338 |       asyncio.staggered
import time:       240 |        240 |       asyncio.trsock
import time:       819 |      10619 |     asyncio.base_events
import time:       554 |        554 |     asyncio.runners
import time:       423 |        423 |     asyncio.queues
import time:       359 |        359 |     asyncio.streams
import time:       317 |        317 |     asyncio.subprocess
import time:       170 |        170 |     asyncio.taskgroups
import time:       325 |        325 |     asyncio.timeouts
import time:       129 |        129 |     asyncio.threads
import time:       240 |        240 |       asyncio.base_subprocess
import time:       557 |        557 |       asyncio.selector_events
import time:       606 |       1402 |     asyncio.unix_events
import time:       318 |      14611 |   asyncio
import time:       258 |        258 |         unittest.util
import time:       255 |        513 |       unittest.result
import time:       569 |        569 |         difflib
import time:       731 |       1300 |       unittest.case
import time:       300 |        300 |       unittest.suite
import time:       531 |        531 |       unittest.loader
import time:       148 |        148 |           unittest.signals
import time:       223 |        371 |         unittest.runner
import time:       223 |        594 |       unittest.main
import time:       234 |       3469 |     unittest
import time:       369 |        369 |     pkgutil
import time:      1629 |       5466 |   unittest.mock
import time:       298 |        298 |   pytest_mock._util
import time:      1267 |      21641 | pytest_mock.plugin
import time:       172 |        172 |   coverage.version
import time:       185 |        185 |     coverage.env
import time:       308 |        308 |           _hashlib
import time:       370 |        370 |           _blake2
import time:       245 |        922 |         hashlib
import time:       282 |        282 |         coverage.exceptions
import time:       606 |        606 |         coverage.misc
import time:       281 |       2089 |       coverage.files
import time:       160 |        160 |       coverage.report
import time:       400 |       2649 |     coverage.annotate
import time:      1450 |       1450 |         configparser
import time:       406 |        406 |         coverage.tomlconfig
import time:       389 |       2244 |       coverage.config
import time:       515 |        515 |       coverage.debug
import time:       225 |        225 |       coverage.disposition
import time:       293 |        293 |       coverage.pytracer
import time:       269 |        269 |       coverage.tracer
import time:       367 |       3910 |     coverage.collector
import time:       179 |        179 |     coverage.context
import time:      1851 |       1851 |             _sqlite3
import time:       395 |       2246 |           sqlite3.dbapi2
import time:       191 |       2436 |         sqlite3
import time:       195 |        195 |         coverage.numbits
import time:       512 |       3142 |       coverage.sqldata
import time:       201 |       3343 |     coverage.data
import time:       231 |        231 |       coverage.results
import time:       223 |        223 |       coverage.templite
import time:       436 |        889 |     coverage.html
import time:       422 |        422 |       sysconfig
import time:       179 |        179 |           coverage.bytecode
import time:       517 |        517 |           coverage.phystokens
import time:       756 |       1451 |         coverage.parser
import time:       227 |        227 |         coverage.plugin
import time:       220 |       1898 |       coverage.python
import time:       309 |       2628 |     coverage.inorout
import time:       171 |        171 |     coverage.jsonreport
import time:       302 |        302 |     coverage.plugin_support
import time:       170 |        170 |     coverage.summary
import time:       274 |        274 |           xml.dom.domreg
import time:       350 |        623 |         xml.dom
import time:       201 |        201 |         xml.dom.minicompat
import time:       137 |        137 |           xml.dom.NodeFilter
import time:       356 |        492 |         xml.dom.xmlbuilder
import time:      1048 |       2364 |       xml.dom.minidom
import time:       242 |       2605 |     coverage.xmlreport
import time:       469 |        469 |           multiprocessing.process
import time:       320 |        320 |               _compat_pickle
import time:       259 |        259 |               _pickle
import time:        90 |         90 |                   org
import time:        15 |        105 |                 org.python
import time:        12 |        117 |               org.python.core
import time:       767 |       1461 |             pickle
import time:       290 |       1751 |           multiprocessing.reduction
import time:       456 |       2675 |         multiprocessing.context
import time:       207 |       2881 |       multiprocessing
import time:       201 |       3081 |     coverage.multiproc
import time:       567 |      20673 |   coverage.control
import time:       179 |      21023 | coverage
import time:        87 |         87 |   StringIO
import time:       317 |        404 | pytest_cov.compat
import time:       329 |        329 |   multiprocessing.util
import time:       376 |        705 | pytest_cov.embed

[webknjaz]

Aha! Adding --no-cov reveals a warning, at least...

$ python -m pytest --no-covImportError while loading conftest '/home/wk/src/github/aio-libs/aiohttp/tests/conftest.py'.
tests/conftest.py:16: in <module>
    from aiohttp.test_utils import loop_context
aiohttp/__init__.py:209: in <module>
    from .worker import GunicornUVLoopWebWorker, GunicornWebWorker
aiohttp/worker.py:11: in <module>
    from gunicorn.config import AccessLogFormat as GunicornAccessLogFormat
/home/wk/.pyenv/versions/aiohttp-pyenv-py3.11.0b5/lib/python3.11/site-packages/gunicorn/config.py:20: in <module>
    from gunicorn import __version__, util
/home/wk/.pyenv/versions/aiohttp-pyenv-py3.11.0b5/lib/python3.11/site-packages/gunicorn/util.py:25: in <module>
    import pkg_resources
/home/wk/.pyenv/versions/aiohttp-pyenv-py3.11.0b5/lib/python3.11/site-packages/pkg_resources/__init__.py:78: in <module>
    __import__('pkg_resources.extern.packaging.requirements')
/home/wk/.pyenv/versions/aiohttp-pyenv-py3.11.0b5/lib/python3.11/site-packages/pkg_resources/_vendor/packaging/requirements.py:9: in <module>
    from pkg_resources.extern.pyparsing import stringStart, stringEnd, originalTextFor, ParseException
/home/wk/.pyenv/versions/aiohttp-pyenv-py3.11.0b5/lib/python3.11/site-packages/pkg_resources/extern/__init__.py:52: in create_module
    return self.load_module(spec.name)
/home/wk/.pyenv/versions/aiohttp-pyenv-py3.11.0b5/lib/python3.11/site-packages/pkg_resources/extern/__init__.py:37: in load_module
    __import__(extant)
/home/wk/.pyenv/versions/aiohttp-pyenv-py3.11.0b5/lib/python3.11/site-packages/pkg_resources/_vendor/pyparsing.py:87: in <module>
    import sre_constants
/home/wk/.pyenv/versions/3.11.0b5/lib/python3.11/sre_constants.py:2: in <module>
    warnings.warn(f"module {__name__!r} is deprecated",
E   DeprecationWarning: module 'sre_constants' is deprecated

[webknjaz]

Looks like I'll need to play with the filterwarnings to get this going further...

[webknjaz]

the pytest invocation produces zero output no matter how much verbosity I request from it and the return code is 1:

Fixed that by upgrading coveragepy.

[webknjaz]

Here's test summary under Python 3.11 on my machine. @Dreamsorcerer if you have a minute to take a look, these are the tests needing immediate action. I hope it's the last blocker here.

========================== short test summary info ===========================
SKIPPED [1] tests/test_connector.py:1953: Proactor Event loop present only in Windows
SKIPPED [1] tests/test_connector.py:1963: Proactor Event loop present only in Windows
SKIPPED [1] tests/test_connector.py:1977: Proactor Event loop present only in Windows
SKIPPED [1] tests/test_connector.py:2120: Proactor Event loop present only in Windows
SKIPPED [1] tests/test_multipart_helpers.py:446: should raise decoding error: %82 is invalid for latin1
SKIPPED [1] tests/test_multipart_helpers.py:455: should raise decoding error: %E4 is invalid for utf-8
SKIPPED [1] tests/test_multipart_helpers.py:510: urllib.parse.unquote is tolerate to standalone % chars
SKIPPED [1] tests/test_multipart_helpers.py:519: urllib.parse.unquote is tolerate to standalone % chars
SKIPPED [1] tests/test_multipart_helpers.py:99: need more smart parser which respects quoted text
SKIPPED [1] tests/test_http_parser.py:888: C based HTTP parser not available
SKIPPED [1] tests/test_proxy_functional.py:403: we need to reconsider how we test this
SKIPPED [1] tests/test_proxy_functional.py:429: we need to reconsider how we test this
SKIPPED [1] tests/test_web_request.py:728: The check is applied in DEBUG mode only
SKIPPED [1] tests/test_web_runner.py:199: Proactor Event loop present only in Windows
SKIPPED [1] tests/test_web_runner.py:211: Proactor Event loop present only in Windows
SKIPPED [1] tests/test_websocket_parser.py:402: Requires Cython
SKIPPED [1] tests/test_websocket_parser.py:417: Requires Cython
SKIPPED [1] tests/test_web_functional.py:1874: C based HTTP parser not available
XFAIL tests/test_client_functional.py::test_broken_connection[pyloop]
XFAIL tests/test_connector.py::test_del_with_scheduled_cleanup[pyloop]
XFAIL tests/test_web_functional.py::test_http10_keep_alive_default[pyloop]
XFAIL tests/test_web_request.py::test_handler_return_type[pyloop]
  see https://github.com/aio-libs/aiohttp/issues/4572
XFAIL tests/test_web_urldispatcher.py::test_decoded_url_match[pyloop-urldecoded_route]
  Regression in v3.7: https://github.com/aio-libs/aiohttp/issues/5621
XFAIL tests/test_web_urldispatcher.py::test_decoded_url_match[pyloop-urldecoded_route_with_regex]
  Regression in v3.7: https://github.com/aio-libs/aiohttp/issues/5621
XFAIL tests/test_web_urldispatcher.py::test_decoded_url_match[pyloop-urlencoded_route]
  Regression in v3.7: https://github.com/aio-libs/aiohttp/issues/5621
XFAIL tests/autobahn/test_autobahn.py::test_client
XFAIL tests/autobahn/test_autobahn.py::test_server
ERROR tests/test_web_sendfile_functional.py::test_static_file_if_match[no_sendfile-pyloop-Fri, 31 Dec 0000 23:59:59 GMT]
ERROR tests/test_web_sendfile_functional.py::test_static_file_if_match_custom_tags[sendfile-pyloop-etags0-200-]
FAILED tests/test_http_parser.py::test_c_parser_loaded - AssertionError: as...
FAILED tests/test_web_app.py::test_appkey_repr_concrete - AssertionError: a...
FAILED tests/test_web_app.py::test_appkey_repr_nonconcrete - AssertionError...
FAILED tests/test_web_app.py::test_appkey_repr_annotated - AssertionError: ...
FAILED tests/test_web_server.py::test_unsupported_upgrade[pyloop]

(https://github.com/aio-libs/aiohttp/runs/7661304595?check_suite_focus=true#step:10:4640)

[webknjaz]

Here's test summary under Python 3.11 on my machine

Oh, it looks like I was running pytest under a wrong Python version that time. Need to re-test.

[webknjaz]

Here's a more relevant list of failures:

FAILED tests/test_client_functional.py::test_timeout_on_reading_headers[pyloop]
FAILED tests/test_client_functional.py::test_timeout_on_conn_reading_headers[pyloop]
FAILED tests/test_loop.py::TestCase::test_default_loop - RuntimeError: Ther...
FAILED tests/test_loop.py::TestCase::test_on_startup_hook - RuntimeError: T...
FAILED tests/test_test_utils.py::TestAioHTTPTestCase::test_example_with_loop
FAILED tests/test_test_utils.py::TestAioHTTPTestCase::test_example_without_explicit_loop
FAILED tests/test_test_utils.py::TestAioHTTPTestCase::test_inner_example - ...
FAILED tests/test_test_utils.py::TestAioHTTPTestCase::test_inner_example_without_explicit_loop

[Dreamsorcerer]

Also visible in the CI run: https://github.com/aio-libs/aiohttp/runs/7661304595?check_suite_focus=true

We're already discussing a PR for the event loop issue, which I think is probably all the last 6 failures. Will just need to look at the 2 test_client_functional.py failures.

[KRRT7]

python 3.11 is now a Release Candidate with Python 3.11.0b5, any chance you guys could get this working for 3.11 now?

[SpaceMonkeyForever]

So there is no way to get this to work on Python3.11.0rc1?

[webknjaz]

python 3.11 is now a Release Candidate with Python 3.11.0b5, any chance you guys could get this working for 3.11 now?

Yes, there is a chance.

So there is no way to get this to work on Python3.11.0rc1?

pip install git+...

[owocado]

@webknjaz wouldn't it be a bit misleading to suggest pip install git+... method currently? since it errors and doesn't work yet on Python 3.11.0rc1.

(py311env) ➜  ~ pip install --no-cache -U git+https://github.com/aio-libs/aiohttp.git
Collecting git+https://github.com/aio-libs/aiohttp.git
  Cloning https://github.com/aio-libs/aiohttp.git to /tmp/pip-req-build-ekcotr_u
  Running command git clone --filter=blob:none --quiet https://github.com/aio-libs/aiohttp.git /tmp/pip-req-build-ekcotr_u
  Resolved https://github.com/aio-libs/aiohttp.git to commit 5bf9b4aae7046704b418c16452c2d1ced934abfb
  Running command git submodule update --init --recursive -q
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting charset-normalizer<3.0,>=2.0
  Downloading charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Requirement already satisfied: multidict<7.0,>=4.5 in ./py311env/lib/python3.11/site-packages (from aiohttp==4.0.0a1) (6.0.2)
Collecting async-timeout<5.0,>=4.0
  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Requirement already satisfied: yarl<2.0,>=1.0 in ./py311env/lib/python3.11/site-packages (from aiohttp==4.0.0a1) (1.8.1)
Collecting typing-extensions>=3.7.4
  Downloading typing_extensions-4.3.0-py3-none-any.whl (25 kB)
Requirement already satisfied: frozenlist>=1.1.1 in ./py311env/lib/python3.11/site-packages (from aiohttp==4.0.0a1) (1.3.1)
Collecting aiosignal>=1.1.2
  Downloading aiosignal-1.2.0-py3-none-any.whl (8.2 kB)
Requirement already satisfied: idna>=2.0 in ./py311env/lib/python3.11/site-packages (from yarl<2.0,>=1.0->aiohttp==4.0.0a1) (3.3)
Building wheels for collected packages: aiohttp
  Building wheel for aiohttp (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for aiohttp (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [88 lines of output]
      *********************
      * Accelerated build *
      *********************
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-cpython-311
      creating build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/client_proto.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/web_runner.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/web_routedef.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/helpers.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/web.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/client_ws.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/locks.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/worker.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/client_reqrep.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/client_exceptions.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/connector.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/typedefs.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/streams.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/tcp_helpers.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/tracing.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/__init__.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/multipart.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/web_server.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/test_utils.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/web_request.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/http_parser.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/base_protocol.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/http_exceptions.py -> build/lib.linux-aarch64-cpython-311/aiohttp      copying aiohttp/http_writer.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/http.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/log.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/resolver.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/web_protocol.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/web_middlewares.py -> build/lib.linux-aarch64-cpython-311/aiohttp      copying aiohttp/pytest_plugin.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/hdrs.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/formdata.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/web_response.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/web_ws.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/web_fileresponse.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/client.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/abc.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/payload.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/http_websocket.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/cookiejar.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/web_log.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/web_app.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/web_urldispatcher.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/web_exceptions.py -> build/lib.linux-aarch64-cpython-311/aiohttp
      running egg_info
      writing aiohttp.egg-info/PKG-INFO
      writing dependency_links to aiohttp.egg-info/dependency_links.txt
      writing requirements to aiohttp.egg-info/requires.txt
      writing top-level names to aiohttp.egg-info/top_level.txt
      reading manifest file 'aiohttp.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching 'aiohttp' anywhere in distribution
      warning: no previously-included files matching '*.pyc' found anywhere in distribution
      warning: no previously-included files matching '*.pyd' found anywhere in distribution
      warning: no previously-included files matching '*.so' found anywhere in distribution
      warning: no previously-included files matching '*.lib' found anywhere in distribution
      warning: no previously-included files matching '*.dll' found anywhere in distribution
      warning: no previously-included files matching '*.a' found anywhere in distribution
      warning: no previously-included files matching '*.obj' found anywhere in distribution
      warning: no previously-included files found matching 'aiohttp/*.html'
      no previously-included directories found matching 'docs/_build'
      adding license file 'LICENSE.txt'
      writing manifest file 'aiohttp.egg-info/SOURCES.txt'
      copying aiohttp/_cparser.pxd -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/_find_header.pxd -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/_helpers.pyi -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/_helpers.pyx -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/_http_parser.pyx -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/_http_writer.pyx -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/_websocket.pyx -> build/lib.linux-aarch64-cpython-311/aiohttp
      copying aiohttp/py.typed -> build/lib.linux-aarch64-cpython-311/aiohttp
      running build_ext
      building 'aiohttp._websocket' extension
      creating build/temp.linux-aarch64-cpython-311
      creating build/temp.linux-aarch64-cpython-311/aiohttp
      aarch64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/ubuntu/py311env/include -I/usr/include/python3.11 -c aiohttp/_websocket.c -o build/temp.linux-aarch64-cpython-311/aiohttp/_websocket.o
      cc1: fatal error: aiohttp/_websocket.c: No such file or directory
      compilation terminated.
      error: command '/usr/bin/aarch64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for aiohttp
Failed to build aiohttp
ERROR: Could not build wheels for aiohttp, which is required to install pyproject.toml-based projects
(py311env) ➜  ~ python -V
Python 3.11.0rc1

[webknjaz]

since it errors and doesn't work yet on Python 3.11.0rc1.

Not really. I forgot to mention that if you want C-extensions, you'd have to clone and run make cythonize. If you don't, AIOHTTP_NO_EXTENSIONS=1 pip install git+.. should work letting you use pure-python fallbacks (that are slower, obviously).

[SpaceMonkeyForever]

@webknjaz I'm sorry, I don't get it. How am I supposed to install this package into my Python env? running make fails with various errors. Last error I got when running make is:

make: *** No rule to make target 'vendor/llhttp/package.json', needed by 'vendor/llhttp/node_modules'. Stop.

If I run: pip install --no-cache -U git+https://github.com/aio-libs/aiohttp.git

It fails with the error @ow0x mentioned

If I run pip install ./aiohttp on the cloned repo where I did make cythonize it also fails with:

aiohttp/_http_parser.c:749:10: fatal error: ../vendor/llhttp/build/llhttp.h: No such file or directory
        749 | #include "../vendor/llhttp/build/llhttp.h"
            |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

I just want to install aiohttp in a python3.11 env.. it's proving to be impossible.

[webknjaz]

It should be make cythonize (creates C-files out of the pyx modules), followed by something like python -m build or pip install . depending on what you want.

[webknjaz]

Looks like make generate-llhttp is needed too, (and maybe synching git submodules?). FWIW check what the CI calls and follow that. It now runs a py3.11 job on master successfully.

[SpaceMonkeyForever]

@webknjaz That seems to have worked, thank you! Btw, make generate-llhttp has this issue:

5 vulnerabilities (3 moderate, 1 high, 1 critical)

To address issues that do not require attention, run:
  npm audit fix

I fixed it by doing cd vendor/llhttp && npm audit fix

[Dreamsorcerer]

I fixed it by doing cd vendor/llhttp && npm audit fix

I suspect we actually just need to update the submodule to the latest release of llhttp.

[SpaceMonkeyForever]

@webknjaz after installing aiohttp from github, I tried installing my projects requirements and it still tried to build and install aiohttp wheel. Why wouldn't it use the already installed aiohttp? It even printed:

Requirement already satisfied: aiohttp>=3.8 in /home/azureuser/pyenvs/env-trader-py311/lib/python3.11/site-packages (from ccxt==1.92.49->-r requirements.txt (line 4)) (4.0.0a1)

But then later it printed:

Building wheels for collected packages: cryptofeed, matplotlib, pandas, pyserum, rxpy-backpressure, ujson, **aiohttp**, yarl, aiofile, construct, jsonrpcclient, jsonrpcserver, order-book, websockets, yapic.json, zstandard, cchardet, uvloop, psutil, pyrsistent

Notice aiohttp was mentioned above. Why would it try to build it again? sorry to bother you with too many questions.

[webknjaz]

Dunno, why aren't you using the 3.8 branch? Master is not ready for public consumption. Maybe because 4.0 is alpha and you didn't use --pre, not did you request a specific alpha release. FWIW it's more of a pip question, then aiohttp.

[ap--]

Hey everyone, in case it might be useful to others, the way I got this working in the CI with Python=3.11 is:

AIOHTTP_NO_EXTENSIONS=1 python -m pip install aiohttp --no-binary aiohttp

Cheers, Andreas :smiley:

[ssbarnea]

That reminds me of pyyaml which automatically fallsback to no-extension when the compilation fails during install. To be it sounds a little bit more resilient,... with the risk that some users will use the considerably slower version, likely without even noticing.

[KRRT7]

i have a question, does using AIOHTTP_NO_EXTENSIONS=1 & --no-binary have any effects on the performance of aiohttp?

[webknjaz]

i have a question, does using AIOHTTP_NO_EXTENSIONS=1 & --no-binary have any effects on the performance of aiohttp?

Yes. With no C-extensions, aiohttp uses pure-Python fallbacks for things that are usually performance bottlenecks. Hence, it'll be slower. Use it for testing/playing around but not for production.

[webknjaz]

So the problem on the 3.9 branch in GHA under Windows seems to be that this fix https://github.com/aio-libs/aiohttp/pull/6756 hasn't been backported. A backport PR for it exists, it just was never merged: https://github.com/aio-libs/aiohttp/pull/6759. Merging now...

[honglei]

Any updates for Python3.11, release a new version on https://pypi.org/project/aiohttp/#files ?

[webknjaz]

Still working on it. Be patient.

[webknjaz]

UPD: 3.9 branch is green now. The last bit necessary was backporting removal of Python 3.5 support that was only applied to the master branch: https://github.com/aio-libs/aiohttp/pull/4046#issuecomment-1242576265. Apparently, async-generator explodes sometimes under Python 3.11. It was still present in the 3.8 and 3.9, forgotten there and we've only started seeing the problems now. Those branches don't support Python 3.5 so it was safe to just cherry-pick that PR.

Next up: making the 3.8 branch green is still needed. The failures seem to be something that's already fixed on master and 3.9 so maybe I'll be lucky to find something to backport too.

[webknjaz]

UPD: 3.8 is green too. I'll now have to see how the release automation is doing and maybe make some changes before cutting the release.

[webknjaz]

UPD: I've started adding new CI jobs. The only one for Python 3.11 was Ubuntu but after adding macos and Windows, new things poped up. That's another blocker for now.