facebook
unable to sl clone wezterm on a mac
I persistently hit too many open files errors during cloning/updating.
I'm running on an M1 macbook air, running macOS Ventura 13.0.1. I installed using brew and the bottle from the download/install instructions.
(the =sl thing here just tells zsh to run the sapling CLI instead of my existing sl shell function that gives me a smartlog output equivalent depending on the repo I'm in)
; =sl clone https://github.com/wez/wezterm.git wezterm-sl
remote: Enumerating objects: 57111, done.
remote: Counting objects: 100% (105/105), done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 57111 (delta 49), reused 67 (delta 35), pack-reused 57006
Receiving objects: 100% (57111/57111), 281.16 MiB | 5.69 MiB/s, done.
Resolving deltas: 100% (39928/39928), done.
From https://github.com/wez/wezterm
* [new ref] 62e2e996da155ff5e922d6c52fd0797f0005cae4 -> remote/main
pulling submodule deps/harfbuzz/harfbuzz
abort: working copy is missing information or corrupt: Too many open files (os error 24)
14:46 took 1m:03s 255 wez@mba ~/wez-personal
; ulimit -n
1048576
I also tried with ulimit -n set to 2048576 without success.
When borked, I tried to resume the checkout, but it seems perma-borked:
; cd wezterm-sl
; sl go main
pulling submodule deps/freetype/libpng
abort: working copy is missing information or corrupt: "/Users/wez/wez-personal/wezterm-sl/.sl/store/gitmodules/0f54928fd0e2a3c3ad07da7da620492ae6da4b11/.sl/store/segments/v1/multimetalog/meta": when reading LogMetadata
in Log::sync
Log.dir = Filesystem("/Users/wez/wez-personal/wezterm-sl/.sl/store/gitmodules/0f54928fd0e2a3c3ad07da7da620492ae6da4b11/.sl/store/segments/v1/multimetalog")
in MultiLog::write_meta
Caused by 1 errors:
- Too many open files (os error 24)
I also tried the stronger version of raising the number of files:
; sudo sysctl kern.maxfiles=1048576
kern.maxfiles: 30720 -> 1048576
but I still get Too many open files errors during cloning.
Note that the repo, when cloned via git, has fewer than 100k files, including build artifacts under the rust targets directory, and the .git directory:
; find . | wc -l
82893
I wonder if the reported error is miscategorized and isn't really about the number of files at all, but some other limit/constraint being shown without sufficient context?
I haven't tried sapling on my linux machine yet, so I don't know whether this is a mac-specific or wezterm-repo-specific issue.
Seems similar to #155 and #175. @mitrandir77 noticed that he has to edit his zshrc and start a new terminal for ulimit to take effect.
Overall it seems strange and I would like to understand what the open handlers are. If you know something like ls -l /proc/self/fd/ in mac let me know!
Well, this is weird. What I pasted above is what I tried yesterday. Since then, I restarted my terminal.
Cloning just now succeeded, without me having to anything explicit with ulimit (see that it is the stock macos value of 256 in the output below), but note that the sysctl I posted above is still in effect:
=sl clone [email protected]:wez/wezterm.git wezterm-sl-2
remote: Enumerating objects: 57117, done.
remote: Counting objects: 100% (111/111), done.
remote: Compressing objects: 100% (64/64), done.
remote: Total 57117 (delta 52), reused 80 (delta 45), pack-reused 57006
Receiving objects: 100% (57117/57117), 281.16 MiB | 10.20 MiB/s, done.
Resolving deltas: 100% (39931/39931), done.
From ssh://github.com/wez/wezterm
* [new ref] 35fd69b0d5ae8d8e2d6e0f4e806b75ef7c5a7faf -> remote/main
pulling submodule deps/harfbuzz/harfbuzz
pulling submodule deps/freetype/libpng
pulling submodule deps/freetype/zlib
pulling submodule deps/freetype/freetype2
pulling submodule deps/freetype/freetype2/subprojects/dlg
1684 files updated, 0 files merged, 0 files removed, 0 files unresolved
18:35 took 59s wez@mba ~/wez-personal
; ulimit -a
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 8176
-c: core file size (blocks) 0
-v: address space (kbytes) unlimited
-l: locked-in-memory size (kbytes) unlimited
-u: processes 1333
-n: file descriptors 256
18:35 wez@mba ~/wez-personal
I've manually edited the output from this bit to focus on the file related tunables:
; sysctl -A | grep file
kern.maxfiles: 1048576
kern.maxfilesperproc: 10240
kern.num_files: 4103
kern.num_files is read-only: it is the total files open in the kernel. kern.maxfiles is the upper bound for that.
I just did this to restore the prior limit:
; sudo sysctl kern.maxfiles=30720
and restarted my terminal, and I can clone successfully, even though ulimit -n reports 256 files as well.
So... the behavior appears to be non-deterministic, or at least, not obviously correlated with these limits :-/
I am going to chalk this up to https://github.com/facebook/sapling/issues/204 and close this.
