RobertCraigie
Pyright Python became flaky in Docker-based CI since 1.1.296
Since we upgraded to v1.1.296 (which AFAIK changed something about how node is packaged), Pyright became flaky in our Docker-based CI. I would estimate that maybe 1 in 10-20 PRs end with an error as below:
Many runs end with this error
This is what often happens when we run pyright backend script tests (the dirs with our Python code) inside a Docker container:
WARNING: there is a new pyright version available (v1.1.296 -> v1.1.301).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
/app/.venv/lib/python3.11/site-packages/nodeenv.py:26: DeprecationWarning: 'pipes' is deprecated and slated for removal in Python 3.13
import pipes
* Install prebuilt node (19.8.1) .Incomplete read while readingfrom https://nodejs.org/download/release/v19.8.1/node-v19.8.1-linux-x64.tar.gz
.
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/app/.venv/lib/python3.11/site-packages/nodeenv.py", line 1519, in <module>
main()
File "/app/.venv/lib/python3.11/site-packages/nodeenv.py", line 1[10](https://github.com/exponential-hq/isometric/actions/runs/4553000054/jobs/8029070958#step:4:11)4, in main
create_environment(env_dir, args)
File "/app/.venv/lib/python3.[11](https://github.com/exponential-hq/isometric/actions/runs/4553000054/jobs/8029070958#step:4:12)/site-packages/nodeenv.py", line 980, in create_environment
install_node(env_dir, src_dir, args)
File "/app/.venv/lib/python3.11/site-packages/nodeenv.py", line 739, in install_node
install_node_wrapped(env_dir, src_dir, args)
File "/app/.venv/lib/python3.11/site-packages/nodeenv.py", line 762, in install_node_wrapped
download_node_src(node_url, src_dir, args)
File "/app/.venv/lib/python3.11/site-packages/nodeenv.py", line 602, in download_node_src
with ctx as archive:
File "/usr/local/lib/python3.11/contextlib.py", line [13](https://github.com/exponential-hq/isometric/actions/runs/4553000054/jobs/8029070958#step:4:14)7, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/nodeenv.py", line 573, in tarfile_open
tf = tarfile.open(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/tarfile.py", line [16](https://github.com/exponential-hq/isometric/actions/runs/4553000054/jobs/8029070958#step:4:17)30, in open
saved_pos = fileobj.tell()
^^^^^^^^^^^^
AttributeError: 'bytes' object has no attribute 'tell'
Traceback (most recent call last):
File "/app/.venv/bin/pyright", line 8, in <module>
sys.exit(entrypoint())
^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/pyright/cli.py", line 34, in entrypoint
sys.exit(main(sys.argv[1:]))
^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/pyright/cli.py", line [19](https://github.com/exponential-hq/isometric/actions/runs/4553000054/jobs/8029070958#step:4:20), in main
return run(*args, **kwargs).returncode
^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/pyright/cli.py", line [25](https://github.com/exponential-hq/isometric/actions/runs/4553000054/jobs/8029070958#step:4:26), in run
pkg_dir = install_pyright(args, quiet=None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/pyright/_utils.py", line 61, in install_pyright
node.run(
File "/app/.venv/lib/python3.11/site-packages/pyright/node.py", line 98, in run
binary = _ensure_available(target)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/pyright/node.py", line [37](https://github.com/exponential-hq/isometric/actions/runs/4553000054/jobs/8029070958#step:4:38), in _ensure_available
return Binary(path=_ensure_node_env(target), strategy=Strategy.NODEENV)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/pyright/node.py", line 64, in _ensure_node_env
_install_node_env()
File "/app/.venv/lib/python3.11/site-packages/pyright/node.py", line 91, in _install_node_env
subprocess.run(args, check=True)
File "/usr/local/lib/python3.11/subprocess.py", line [57](https://github.com/exponential-hq/isometric/actions/runs/4553000054/jobs/8029070958#step:4:58)1, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/app/.venv/bin/python', '-m', 'nodeenv', '/github/home/.cache/pyright-python/nodeenv']' returned non-zero exit status 1.
Here's the Dockerfile for reference
pyright-python is a dev group dependency in the pyproject.toml file:
# These values should be overriden at image build time
ARG COMMIT_HASH=dev-docker
FROM python:3.11.1 as python
ENV PYTHONUNBUFFERED=true
WORKDIR /app
FROM python:3.11.1-slim AS python-slim
ENV PYTHONUNBUFFERED=true
WORKDIR /app
# Base environment with Poetry
FROM python AS poetry
# Install Poetry using the official install script, to isolate it from application dependencies
ENV POETRY_VERSION=1.4.0
ENV POETRY_HOME=/opt/poetry
RUN curl -sSL https://install.python-poetry.org | python - --version $POETRY_VERSION
ENV PATH="$POETRY_HOME/bin:$PATH"
# Install the virtualenv inside the project to make it easy to copy over the dependencies later
ENV POETRY_VIRTUALENVS_IN_PROJECT=true
# Stage to build a virtual environment with Python production dependencies
FROM poetry AS prod-venv
# Install the main dependencies (before any source code files are copied)
COPY pyproject.toml poetry.lock poetry.toml ./
RUN poetry install -vvv --no-interaction --no-ansi --no-root --only=main
# Environment for running various development actions (tests, migrations, etc.)
FROM prod-venv AS dev
# Additionally install the dev dependencies (before any source code files are copied)
RUN poetry install -vvv --no-interaction --no-ansi --no-root
# Print the version to actually install Pyright, as the pyright-python project does a lazy installation
RUN poetry run pyright --version
# Copy source code files and folders necessary for running static checks
COPY alembic.ini schema.graphql ./
COPY backend ./backend
COPY scripts ./scripts
COPY tests ./tests
# Install the application itself properly (to make sure that 'backend' is in the path)
RUN poetry install -vvv --no-interaction --no-ansi
# Make the entrypoint to this stage running a Poetry task
ENTRYPOINT ["poetry", "run", "task"]
I am not sure if it's good or bad, but the Docker image itself seems to be fine. I.e. if we retry running Pyright using the same image, it might just work!
Any ideas for what could be causing this @RobertCraigie ? Or is there anything else I can do to help debug this?
Since the image seems to be fine (e.g. it's possible to create a new container from the same image and Pyright might run), I'll try to add retries in this way:
for i in {1..3}; do pyright backend tests scripts && break; done
Though there may be something deterministic about a specific container that stops Pyright from running. I shall see!
Just looking through the CI logs, I also found instances where the operation just timed out (total CI run reached 15 minutes):
Interesting, I have seen that AttributeError: 'bytes' object has no attribute 'tell' error before but it was very rare so I assumed it was an ephemeral network issue, but if you're encountering this frequently there must be something else going on...
I do remember seeing this error before v1.1.296 so I'm not sure it's related but it's weird that it only started happening recently for you.
Maybe you could try installing nodejs-bin beforehand as an alternative node.js installer?
pip install 'nodejs-bin[cmd]'
I'm planning on adding native support for nodejs-bin as well.
Thanks, will give it a try!
Maybe it happened before too, it just felt like something that started recently...
FWIW GitHub workflows do have network issues occasionally. But note that these problems happened at container runtime, not image build time. And I do print the version of Pyright at image build time -- to ensure that node is baked into the image.
So it doesn't look like a network issue, because the node dependency should have been already baked in...
I have this issue too. I think it is a network failure, even if you already initialised pre-commit. I think the issue is that the pre-commit install hook doesn't actually install Pyright. So it is downloaded every time you run pre-commit.
I guess a workaround would be to run it once as part of the Docker build but it would be nice if this was fixed.
Also it appears to access the network to check for a newer version of Pyright even if it is already installed. From the code it looks like you can export PYRIGHT_PYTHON_IGNORE_WARNINGS=1 to avoid that.
I am also suffering from this intermittent issue. I have some symptom detail to add.
- I can reproduce the failure 70% of attempts. For test convenience, I am using a docker container based on python:3.9-slim running in windows WSL2. I am using poetry to lock down my dev dependency versions including pyright.
- It does appear to be a slow/unresponsive repository server when the lazy install of node/pyright.js is taking place. Looking at the network graph I can predict when its going to fail long before the network timeout. I suspect there are multiple remote servers to choose from and few are giving a generous bandwidth or connection at all. But the lazy installer is simplistic and not hunting round for the best server.
from pkg_resources import parse_version
* Install prebuilt node (20.5.0)
I am now doubting the convenient installation of pyright via poetry or pip is the right option for a CI pipeline as it needs to be reliable. I would rather it fail at the install of pyright rather than when its first used to test my app. I have considered basing my CI build image on node to make pyright more reliable and then add my python stuff on top but this is also not how I want my CI.
@beaumsc I completely agree with your considerations. Have you found a solution you liked?
Any updates on this? I'm also facing an issue where my pyright command times out while installing pyright with poetry works fine.
#22 152.2 Traceback (most recent call last):
#22 152.2 File "/usr/local/lib/python3.12/urllib/request.py", line 1344, in do_open
#22 152.2 h.request(req.get_method(), req.selector, req.data, headers,
#22 152.2 File "/usr/local/lib/python3.12/http/client.py", line 1327, in request
#22 152.2 self._send_request(method, url, body, headers, encode_chunked)
#22 152.2 File "/usr/local/lib/python3.12/http/client.py", line 1373, in _send_request
#22 152.2 self.endheaders(body, encode_chunked=encode_chunked)
#22 152.2 File "/usr/local/lib/python3.12/http/client.py", line 1322, in endheaders
#22 152.2 self._send_output(message_body, encode_chunked=encode_chunked)
#22 152.2 File "/usr/local/lib/python3.12/http/client.py", line 1081, in _send_output
#22 152.2 self.send(msg)
#22 152.2 File "/usr/local/lib/python3.12/http/client.py", line 1025, in send
#22 152.2 self.connect()
#22 152.2 File "/usr/local/lib/python3.12/http/client.py", line 1461, in connect
#22 152.2 super().connect()
#22 152.2 File "/usr/local/lib/python3.12/http/client.py", line 991, in connect
#22 152.2 self.sock = self._create_connection(
#22 152.2 ^^^^^^^^^^^^^^^^^^^^^^^^
#22 152.2 File "/usr/local/lib/python3.12/socket.py", line 852, in create_connection
#22 152.2 raise exceptions[0]
#22 152.2 File "/usr/local/lib/python3.12/socket.py", line 837, in create_connection
#22 152.2 sock.connect(sa)
#22 152.2 OSError: [Errno 99] Cannot assign requested address
#22 152.2
#22 152.2 During handling of the above exception, another exception occurred:
#22 152.2
#22 152.2 Traceback (most recent call last):
#22 152.2 File "<frozen runpy>", line 198, in _run_module_as_main
#22 152.2 File "<frozen runpy>", line 88, in _run_code
#22 152.2 File "/app/.venv/lib/python3.12/site-packages/nodeenv.py", line 1548, in <module>
#22 152.2 main()
#22 152.2 File "/app/.venv/lib/python3.12/site-packages/nodeenv.py", line 1119, in main
#22 152.2 args.node = get_last_stable_node_version()
#22 152.2 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#22 152.2 File "/app/.venv/lib/python3.12/site-packages/nodeenv.py", line 1052, in get_last_stable_node_version
#22 152.2 return _get_versions_json()[0]['version'].lstrip('v')
#22 152.2 ^^^^^^^^^^^^^^^^^^^^
#22 152.2 File "/app/.venv/lib/python3.12/site-packages/nodeenv.py", line 1028, in _get_versions_json
#22 152.2 response = urlopen('%s/index.json' % src_base_url)
#22 152.2 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#22 152.2 File "/app/.venv/lib/python3.12/site-packages/nodeenv.py", line 652, in urlopen
#22 152.2 return urllib2.urlopen(req)
#22 152.2 ^^^^^^^^^^^^^^^^^^^^
#22 152.2 File "/usr/local/lib/python3.12/urllib/request.py", line 215, in urlopen
#22 152.2 return opener.open(url, data, timeout)
#22 152.2 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#22 152.2 File "/usr/local/lib/python3.12/urllib/request.py", line 515, in open
#22 152.2 response = self._open(req, data)
#22 152.2 ^^^^^^^^^^^^^^^^^^^^^
#22 152.2 File "/usr/local/lib/python3.12/urllib/request.py", line 532, in _open
#22 152.2 result = self._call_chain(self.handle_open, protocol, protocol +
#22 152.2 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#22 152.2 File "/usr/local/lib/python3.12/urllib/request.py", line 492, in _call_chain
#22 152.2 result = func(*args)
#22 152.2 ^^^^^^^^^^^
#22 152.2 File "/usr/local/lib/python3.12/urllib/request.py", line 1392, in https_open
#22 152.2 return self.do_open(http.client.HTTPSConnection, req,
#22 152.2 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#22 152.2 File "/usr/local/lib/python3.12/urllib/request.py", line 1347, in do_open
#22 152.2 raise URLError(err)
#22 152.2 urllib.error.URLError: <urlopen error [Errno 99] Cannot assign requested address>
#22 152.2 Traceback (most recent call last):
#22 152.2 File "/app/.venv/bin/pyright", line 8, in <module>
#22 152.2 sys.exit(entrypoint())
#22 152.2 ^^^^^^^^^^^^
#22 152.2 File "/app/.venv/lib/python3.12/site-packages/pyright/cli.py", line 34, in entrypoint
#22 152.2 sys.exit(main(sys.argv[1:]))
#22 152.2 ^^^^^^^^^^^^^^^^^^
#22 152.2 File "/app/.venv/lib/python3.12/site-packages/pyright/cli.py", line 19, in main
#22 152.2 return run(*args, **kwargs).returncode
#22 152.2 ^^^^^^^^^^^^^^^^^^^^
#22 152.2 File "/app/.venv/lib/python3.12/site-packages/pyright/cli.py", line 25, in run
#22 152.2 pkg_dir = install_pyright(args, quiet=None)
#22 152.2 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#22 152.2 File "/app/.venv/lib/python3.12/site-packages/pyright/_utils.py", line 64, in install_pyright
#22 152.2 node.run(
#22 152.2 File "/app/.venv/lib/python3.12/site-packages/pyright/node.py", line 105, in run
#22 152.2 binary = _ensure_available(target)
#22 152.2 ^^^^^^^^^^^^^^^^^^^^^^^^^
#22 152.2 File "/app/.venv/lib/python3.12/site-packages/pyright/node.py", line 38, in _ensure_available
#22 152.2 return Binary(path=_ensure_node_env(target), strategy=Strategy.NODEENV)
#22 152.2 ^^^^^^^^^^^^^^^^^^^^^^^^
#22 152.2 File "/app/.venv/lib/python3.12/site-packages/pyright/node.py", line 65, in _ensure_node_env
#22 152.2 _install_node_env()
#22 152.2 File "/app/.venv/lib/python3.12/site-packages/pyright/node.py", line 98, in _install_node_env
#22 152.2 subprocess.run(args, check=True)
#22 152.2 File "/usr/local/lib/python3.12/subprocess.py", line 571, in run
#22 152.2 raise CalledProcessError(retcode, process.args,
#22 152.2 subprocess.CalledProcessError: Command '['/app/.venv/bin/python', '-m', 'nodeenv', '/root/.cache/pyright-python/nodeenv']' returned non-zero exit status 1.
#22 ERROR: process "/bin/sh -c poetry run pyright" did not complete successfully: exit code: 1
------
> [tests 8/9] RUN poetry run pyright:
152.2 File "/app/.venv/lib/python3.12/site-packages/pyright/node.py", line 38, in _ensure_available
152.2 return Binary(path=_ensure_node_env(target), strategy=Strategy.NODEENV)
152.2 ^^^^^^^^^^^^^^^^^^^^^^^^
152.2 File "/app/.venv/lib/python3.12/site-packages/pyright/node.py", line 65, in _ensure_node_env
152.2 _install_node_env()
152.2 File "/app/.venv/lib/python3.12/site-packages/pyright/node.py", line 98, in _install_node_env
152.2 subprocess.run(args, check=True)
152.2 File "/usr/local/lib/python3.12/subprocess.py", line 571, in run
152.2 raise CalledProcessError(retcode, process.args,
152.2 subprocess.CalledProcessError: Command '['/app/.venv/bin/python', '-m', 'nodeenv', '/root/.cache/pyright-python/nodeenv']' returned non-zero exit status 1.
