pixi
pixi copied to clipboard
prefix-dev
Too many open files
Checks
-
[X] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pixi, using
pixi --version.
Reproducible example
[project]
...
channels = ["conda-forge"]
platforms = ["linux-64", "linux-aarch64", "osx-arm64"]
[dependencies]
beautifulsoup4 = ">=4,<5"
chromadb = "<1"
coverage = ">=7.4.1,<7.5"
cyclonedx-bom = ">=3.11.7,<3.12"
datamodel-code-generator = ">=0.25.4,<0.26"
dateparser = ">=1.2.0,<1.3"
easyocr = ">=1.7.1,<1.8"
fastapi = ">=0.103.0,<0.104"
httpx = ">=0.26.0,<0.27"
ipython = ">=8.21.0,<8.22"
jupyter = ">=1.0.0,<1.1"
lxml = ">=5.1.0,<5.2"
openai = ">=1.35.3,<1.36"
openpyxl = ">=3.1.2,<3.2"
pandas = "2.*"
passlib = ">=1.7.4,<1.8"
pdbpp = ">=0.10.3,<0.11"
pdf2image = "1.16.*"
pixi-pycharm = ">=0.0.5,<0.1"
polars = ">=0.20.23,<0.21"
py-spy = ">=0.3.14,<0.4"
py7zr = ">=0.20.8,<0.21"
pydantic = ">=2.6.1,<2.7"
pyexcel = ">=0.7.0,<0.8"
pyexcel-xls = ">=0.6.2,<0.7"
pyexcel-xlsx = ">=0.6.0,<0.7"
pymongo = ">=4.6.1,<4.7"
pypdf = ">=4.0.2,<4.1"
pytesseract = ">=0.3.13,<0.4"
pytest = "<8" # https://github.com/TvoroG/pytest-lazy-fixture/issues/65
python = ">=3.12"
python-dotenv = ">=1.0.1,<1.1"
python-jose = ">=3.3.0,<3.4"
python-multipart = ">=0.0.7,<0.1"
sentence-transformers = ">=2.6.1,<2.7"
tabulate = ">=0.9.0,<0.10"
tesseract = ">=5.3.4,<5.4"
testcontainers = ">=4.4.0,<5"
thefuzz = ">=0.22.1,<0.23"
tiktoken = ">=0.6.0,<0.7"
transformers = ">=4.39.2,<4.40"
uvicorn = ">=0.27.0.post1,<0.27.1"
xmlschema = ">=3.2.0,<3.3"
xmltodict = ">=0.13.0,<0.14"
python-xxhash = ">=3.4.1,<3.5"
pyarrow = "<16" # pandas incompat
requests = "<2.32" # https://github.com/docker/docker-py/issues/3256
pytorch = "<2.3"
pytest-lazy-fixture = ">=0.6.3,<0.7"
loguru = ">=0.7.2,<0.8"
python-docx = ">=1.1.2,<1.2"
markdown = ">=3.6,<4"
pytest-xdist = ">=3.6.1,<3.7"
waitress = ">=3.0.0,<3.1"
pyright = ">=1.1.368,<1.2"
Issue description
Running a large new environment for the first time, on macOS:
❯ pixi run python
⠂ default:linux-aarch6400:00:03] resolving conda
⠂ default:linux-64 [00:00:03] loading repodata
⠙ validate cache [00:00:00] [━━━━━━━━━━━━━━──────] tesseract (+167)
⠙ download & extract [00:00:00] [────────────────────] 0 B @ 0 B/s scikit-learn (+100)
⠙ installing packages [00:00:00] [────────────────────] 0/536 sentencepiece-python (+161) × failed to instantiate a prefix for 'default'
├─▶ failed to link xmltodict-0.13.0-pyhd8ed1ab_0.tar.bz2
├─▶ failed to read 'index.json'
╰─▶ Too many open files (os error 24)
❯ wc -l pixi.toml
105 pixi.toml
❯ wc -l pixi.lock
25153 pixi.lock
❯ grep platform pixi.toml
platforms = ["linux-64", "linux-aarch64", "osx-arm64"]
Expected behavior
Should do fewer things in parallel so as to not go over macOS ulimit -n
While I agree that this can be a problem other package managers also frequently run into it (e.g. Yarn).
We could try to figure out the current ulimit though and try to stay below it. Might be a bit of refactor though in rattler ...
Maybe an easier workaround would be to improve the error message. Maybe you can globally hardcode something like
if error message contains "Too many open files" or errno = 24
print "Trying running ulimit -n 12345"
Mm, we do have some code in place to ensure we don't perform too many IO requests. Do you always get the error failed to link .., or is it sometimes something else? Also, what is your current ulimit?
macOS default: 256
I was unable to reproduce just now. I'll post the error messages when it happens again
Got it again:
❯ mv .pixi/ pixi
❯ pixi run ipython
⠁ validate cache [00:00:00] [━╾──────────────────] tesseract (+521)
⠁ download & extract [00:00:00] [────────────────────] 0 B @ 0 B/s libhwloc (+10)
⠁ installing packages [00:00:00] [────────────────────] 0/569 libopenvino-auto-batch-plugin (+28) × failed to link pkgconfig-1.5.5-pyhd8ed1ab_4.tar.bz2
├─▶ failed to read 'link.json'
╰─▶ Too many open files (os error 24)
❯ ulimit -n
256
I have the same ulimit, but don't see this error often. Your environment does look pretty big though (569 packages).
I wonder if we could defend against this and react on the specific "too many open files" error by waiting, and retrying a few milliseconds later.
I also get this error with 0.27.0 on MacOS with default
$ ulimit -n
256
Followed this guide: https://holoviz-dev.github.io/panel/developer_guide/index.html
And get this error:
(base) panel $ pixi run test-unit
✨ Pixi task (test-unit in test-312): pytest panel/tests -n logical --dist loadgroup
⠁ validate cache [00:00:00] [━╾──────────────────] pillow (+298)
⠁ download & extract [00:00:00] [────────────────────] 0 B @ 0 B/s qt6-main (+133)
⠁ installing packages [00:00:00] [────────────────────] 0/452 brotli (+4) × failed to link brotli-bin-1.1.0-hb547adb_1.conda
├─▶ failed to read 'paths.json'
╰─▶ Too many open files (os error 24)
Would be great if you could improve pixi to make this work with defaults on MacOS.
Anyone knows what the workaround is? With Google I find various commands to change system settings for older versions of MacOS but usually with comments that they don't work reliably or are not recommended.
Bas' PR works for me on macOS so I think we can merge and it'll be in the next release :)
@wolfv @baszalmstra I'm still seeing this on Apple silicon macOS on v0.28.2 with pixi global unless I manually expand with ulimit -n
~ % pixi self-update
✔ pixi is already up-to-date (version 0.28.2)
~ % pixi global upgrade-all
× failed to parse '/Users/feickert/.pixi/envs/uv/conda-meta/uv-0.4.2-hd3a8144_0.json'
╰─▶ Too many open files (os error 24)
~ % ulimit -n
256
~ % ulimit -n 2048
~ % pixi global upgrade-all
Nothing to upgrade
This is happening with a decently large pixi global collection
pixi global list:
~ % pixi global list
Global install location: /Users/feickert/.pixi
├── anaconda-client 1.12.3
| └─ exec: conda-server, anaconda, binstar
├── asciinema 2.4.0
| └─ exec: asciinema
├── bat 0.24.0
| └─ exec: bat
├── cmake 3.30.3
| └─ exec: cmake, ctest, cpack, ccmake
├── cookiecutter 2.6.0
| └─ exec: cookiecutter
├── crane 0.20.2
| └─ exec: crane
├── curl 8.9.1
| └─ exec: curl
├── git 2.46.0
| └─ exec: scalar, git-upload-pack, git-shell, git, git-cvsserver, git-credential-osxkeychain, git-upload-archive, gitk, git-receive-pack
├── grayskull 2.7.1
| └─ exec: conda-grayskull, conda-greyskull, grayskull, greyskull
├── hepconvert 1.3.7
| └─ exec: hepconvert
├── hyperfine 1.18.0
| └─ exec: hyperfine
├── jupyter-repo2docker 2024.7.0
| └─ exec: repo2docker, jupyter-repo2docker
├── lhapdf 6.5.4
| └─ exec: lhapdf, lhapdf-config
├── miller 6.12.0
| └─ exec: mlr
├── ninja 1.12.1
| └─ exec: ninja
├── pandoc 3.3
| └─ exec: pandoc-server, pandoc, pandoc-lua
├── pipx 1.7.1
| └─ exec: pipx
├── pre-commit 3.8.0
| └─ exec: pre-commit
├── pytube 15.0.0
| └─ exec: pytube
├── rattler-build 0.20.0
| └─ exec: rattler-build
├── ruff 0.6.2
| └─ exec: ruff
├── shellcheck 0.10.0
| └─ exec: shellcheck
├── tree 2.1.3
| └─ exec: tree
├── uproot-browser 0.6.7
| └─ exec: uproot-browser
├── uv 0.4.2
| └─ exec: uvx, uv
├── visidata 2.11
| └─ exec: vd, visidata
└── xrootd 5.7.0
└─ exec: frm_admin, frm_xfragent, xrdcopy, xrdmapc, xrdcrc32c, xrdgsiproxy, xrdpinls, xrdpwdadmin, xrdreplay, wait41, xrootd, xrdcks, xrdcp, xrdpfc_print, xrdfs, mpxstats, xrdacctest, xrdgsitest, xrdsssadmin, xrootd-config, cmsd, frm_xfrd, frm_purged, cconfig, xrdadler32
I'm not sure how much of this is a shortcoming of macOS defaults vs. something that is fixable on the software side.
Do you see anything related to r limit in the logs when you run it with -v ?
If not would you be able to open a new issue? This may be related to the upgrade-all subcommand.
@baszalmstra I'm not always able to reproduce it — it seems to come and go. I'll be away from the Apple silicon macOS machine this week but I'll try to get a consistent reproducer and make a new issue if needed.
I get this error with pixi 0.28.2 now:
(base) panel $ pixi add pixi-pycharm -v
INFO pixi::environment: verifying prefix location is unchanged, with prefix file: /Users/cdeil/code/oss/panel/.pixi/envs/default/conda-meta/pixi_env_prefix
INFO pixi::lock_file::outdated: the dependencies of environment 'default' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'default' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'default' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'default' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-310' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-310' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-310' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-310' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-311' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-311' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-311' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-311' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-312' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-312' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-312' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-312' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-ui' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-ui' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-ui' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-ui' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-core' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-core' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-core' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-core' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-type' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-type' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-type' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'test-type' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'docs' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'docs' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'docs' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'docs' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'build' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'build' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'build' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'build' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'lint' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'lint' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'lint' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'lint' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'lite' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'lite' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'lite' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::outdated: the dependencies of environment 'lite' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
INFO pixi::lock_file::resolve::uv_resolution_context: uv keyring provider is disabled
WARN rattler_repodata_gateway::fetch: could not compute BLAKE2 hash of repodata.json file: Too many open files (os error 24). Ignoring cached files...
WARN rattler_repodata_gateway::fetch: could not compute BLAKE2 hash of repodata.json file: Too many open files (os error 24). Ignoring cached files...
WARN rattler_repodata_gateway::fetch: could not compute BLAKE2 hash of repodata.json file: Too many open files (os error 24). Ignoring cached files...
⠁ test-ui:osx-arm64 [00:00:00] loading repodata
⠁ test-ui:linux-64 [00:00:00] [━━━━━━━━━━━━━━━━━━━━] 0 B @ 0 B/s downloading repodata
⠁ lite:osx-arm64 [00:00:00] [━━━━━━━━━━━━━━━━━━━━] 0 B @ 0 B/s downloading repodata
⠁ test-ui:osx-64 [00:00:00] [━━━━━━━━━━━━━━━━━━━━] 0 B @ 0 B/s downloading repodata
⠁ build:osx-arm64 [00:00:00] loading repodata
⠁ lite:win-64 [00:00:00] [━━━━━━━━━━━━━━━━━━━━] 0 B @ 0 B/s downloading repodata
⠁ lite:osx-64 [00:00:00] [━━━━━━━━━━━━━━━━━━━━] 0 B @ 0 B/s downloading repodata
⠁ build:win-64 [00:00:00] loading repodata
⠁ lite:linux-64 [00:00:00] [━━━━━━━━━━━━━━━━━━━━] 0 B @ 0 B/s downloading repodata
⠁ test-ui:win-64 [00:00:00] [━━━━━━━━━━━━━━━━━━━━] 0 B @ 0 B/s downloading repodata × failed to instantiate a prefix for 'test-ui'
├─▶ failed to parse '/Users/cdeil/code/oss/panel/.pixi/envs/test-ui/conda-meta/tinycss2-1.3.0-pyhd8ed1ab_0.json'
╰─▶ Too many open files (os error 24)
If not would you be able to open a new issue? This may be related to the upgrade-all subcommand.
I've opened Issue https://github.com/prefix-dev/pixi/issues/2005 for this now. 👍
Just came across this. ulimit -n 10000; pixi install helped here.
May be worth having an FAQ page in the docs to make this more discoverable.
Just came across this.
ulimit -n 10000; pixi installhelped here.May be worth having an FAQ page in the docs to make this more discoverable.
@raybellwaves what version of pixi are you using? I believe this has been resolved since v0.34.0. https://github.com/prefix-dev/pixi/issues/2005#issuecomment-2445908006
Just came across this.
ulimit -n 10000; pixi installhelped here. May be worth having an FAQ page in the docs to make this more discoverable.@raybellwaves what version of
pixiare you using? I believe this has been resolved sincev0.34.0. #2005 (comment)
0.35.0
I don't think I can reproduce my use case as i'm using artifactory and have some packages I build kept there but some have lots of deps
I noticed the default was 256 for ulimit -n