sapling
sapling copied to clipboard
Published
20 hours ago •
facebook
facebook
Error cloning repos with submodules
Hi there!
First, thanks for this great tool!
I have a repository which has a few git submodules. When I clone the repository, the submodules seem to want to be cloned automatically (I actually prefer that, so I'm fine with that). The issue is only the first submodule gets cloned, and an error comes up
pulling submodule forge-std
abort: working copy is missing information or corrupt: "/Users/dteiml/p/hacker-dom/sapling-git-submodules-sl/.sl/store/gitmodules/c40f5e035560142833917ec851ed36f37d9ab2ad/.sl/store/segments/v1/iddag/log": cannot open for mmap
in log::OpenOptions::open(SharedMeta { path: Filesystem("/Users/dteiml/p/hacker-dom/sapling-git-submodules-sl/.sl/store/gitmodules/c40f5e035560142833917ec851ed36f37d9ab2ad/.sl/store/segments/v1/iddag"), meta: Mutex { data: LogMetadata { primary_len: 12, indexes: {}, epoch: 15682642694474209563 }, poisoned: false, .. } })
in multi::OpenOptions::open
Caused by 1 errors:
- Too many open files (os error 24)
I made a repo to illustrate this
^ note that the submodule openzeppelin-contracts is missing from here
Context
That's a ulimit -n issue. By default, MacOS sets the max open file limit to 256 files. You can raise that for your current process by doing ulimit -n <nfiles>. Try ulimit -n 8192 and see if the clone succeeds.
You might need to set ulimit on your .bash_profile or equivalent. Please check this documentation and see if it helps.
sometimes ulimit doesn't help:
> sl clone https://github.com/ClickHouse/ClickHouse.git
remote: Enumerating objects: 916419, done.
remote: Counting objects: 100% (156/156), done.
remote: Compressing objects: 100% (125/125), done.
remote: Total 916419 (delta 72), reused 88 (delta 31), pack-reused 916263
Receiving objects: 100% (916419/916419), 772.35 MiB | 20.69 MiB/s, done.
Resolving deltas: 100% (724758/724758), done.
From https://github.com/ClickHouse/ClickHouse
* [new ref] d05223e70b50968964a568c55b509e0ae4904a8b -> remote/master
pulling submodule contrib/poco
pulling submodule contrib/zstd
pulling submodule contrib/lz4
pulling submodule contrib/librdkafka
pulling submodule contrib/cctz
warning: failed to inspect working copy parent
** Sapling SCM (version 20221116.203146.2c1a971a) has crashed:
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/edenscm/__init__.py", line 106, in run
dispatch.run(args, fin, fout, ferr, config)
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 161, in run
status = (dispatch(req) or 0) & 255
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 503, in dispatch
ret = _runcatch(req)
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 712, in _runcatch
return _callcatch(ui, req, _runcatchfunc)
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 721, in _callcatch
return scmutil.callcatch(ui, req, func)
File "/usr/lib/python3.10/site-packages/edenscm/scmutil.py", line 147, in callcatch
return func()
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 683, in _runcatchfunc
return _dispatch(req)
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 1218, in _dispatch
ret = runcommand(
File "/usr/lib/python3.10/site-packages/edenscm/ext/undo.py", line 106, in _runcommandwrapper
return orig(lui, repo, cmd, fullargs, *args)
File "/usr/lib/python3.10/site-packages/edenscm/ext/journal.py", line 80, in runcommand
return orig(lui, repo, cmd, fullargs, *args)
File "/usr/lib/python3.10/site-packages/edenscm/ext/sparse.py", line 504, in _tracktelemetry
res = runcommand(lui, repo, *args)
File "/usr/lib/python3.10/site-packages/edenscm/ext/copytrace.py", line 173, in _runcommand
return orig(lui, repo, cmd, fullargs, ui, *args, **kwargs)
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 913, in runcommand
ret = _runcommand(ui, options, cmd, d)
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 1259, in _runcommand
return cmdfunc()
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 1217, in <lambda>
d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
File "/usr/lib/python3.10/site-packages/edenscm/util.py", line 1285, in check
return func(*args, **kwargs)
File "/usr/lib/python3.10/site-packages/edenscm/util.py", line 1285, in check
return func(*args, **kwargs)
File "/usr/lib/python3.10/site-packages/edenscm/ext/sparse.py", line 595, in _clonesparsecmd
return orig(ui, repo, *args, **opts)
File "/usr/lib/python3.10/site-packages/edenscm/util.py", line 1285, in check
return func(*args, **kwargs)
File "/usr/lib/python3.10/site-packages/edenscm/util.py", line 1285, in check
return func(*args, **kwargs)
File "/usr/lib/python3.10/site-packages/edenscm/ext/remotefilelog/__init__.py", line 325, in cloneshallow
return orig(ui, source, *args, **opts)
File "/usr/lib/python3.10/site-packages/edenscm/util.py", line 1285, in check
return func(*args, **kwargs)
File "/usr/lib/python3.10/site-packages/edenscm/commands/__init__.py", line 1588, in clone
r = git.clone(ui, giturl, dest, update)
File "/usr/lib/python3.10/site-packages/edenscm/git.py", line 137, in clone
hg.updatetotally(repo.ui, repo, node, None)
File "/usr/lib/python3.10/site-packages/edenscm/hg.py", line 972, in updatetotally
hasunresolved = _update(repo, checkout, updatecheck=updatecheck)
File "/usr/lib/python3.10/site-packages/edenscm/hg.py", line 900, in update
stats = updaterepo(repo, node, False, updatecheck=updatecheck)
File "/usr/lib/python3.10/site-packages/edenscm/ext/remotenames.py", line 327, in exupdate
res = orig(repo, *args, **opts)
File "/usr/lib/python3.10/site-packages/edenscm/hg.py", line 885, in updaterepo
return mergemod.update(
File "/usr/lib/python3.10/site-packages/edenscm/ext/sparse.py", line 419, in _update
results = orig(repo, node, branchmerge, *args, **kwargs)
File "/usr/lib/python3.10/site-packages/edenscm/util.py", line 4197, in inner
return func(*args, **kwargs)
File "/usr/lib/python3.10/site-packages/edenscm/merge.py", line 2440, in update
git.submodulecheckout(p2, matcher, force=force)
File "/usr/lib/python3.10/site-packages/edenscm/git.py", line 584, in submodulecheckout
submod.checkout(node, force=force)
File "/usr/lib/python3.10/site-packages/edenscm/git.py", line 715, in checkout
self.pullnode(repo, node)
File "/usr/lib/python3.10/site-packages/edenscm/git.py", line 699, in pullnode
if node not in repo:
File "/usr/lib/python3.10/site-packages/edenscm/localrepo.py", line 1360, in __contains__
self[changeid]
File "/usr/lib/python3.10/site-packages/edenscm/localrepo.py", line 1346, in __getitem__
return context.changectx(self, changeid)
File "/usr/lib/python3.10/site-packages/edenscm/context.py", line 440, in __init__
repo.changelog.rev(changeid)
File "/usr/lib/python3.10/site-packages/edenscm/localrepo.py", line 95, in __get__
return super(_basefilecache, self).__get__(repo, type)
File "/usr/lib/python3.10/site-packages/edenscm/scmutil.py", line 1181, in __get__
entry.obj = self.func(obj)
File "/usr/lib/python3.10/site-packages/edenscm/localrepo.py", line 1201, in changelog
cl = _openchangelog(self)
File "/usr/lib/python3.10/site-packages/edenscm/localrepo.py", line 3328, in _openchangelog
return changelog2.changelog.opengitsegments(repo, repo.ui.uiconfig())
File "/usr/lib/python3.10/site-packages/edenscm/changelog2.py", line 146, in opengitsegments
inner = bindings.dag.commits.opengitsegments(gitdir, segmentsdir, metalog)
error.RustError: "/home/nyaapa/ClickHouse/.sl/store/gitmodules/3d9d566440b3fd12471707f6d44b412a39d2a8dd/.sl/store/segments/v1/multimetalog/log": cannot open for mmap
in Log::sync
Log.dir = Filesystem("/home/nyaapa/ClickHouse/.sl/store/gitmodules/3d9d566440b3fd12471707f6d44b412a39d2a8dd/.sl/store/segments/v1/multimetalog")
in MultiLog::write_meta
Caused by 1 errors:
- Too many open files (os error 24)
I tried to reproduce, but I get a different error:
jeremy@Jeremys-MacBook-Pro ~ % sl clone https://github.com/ClickHouse/ClickHouse.git
remote: Enumerating objects: 916419, done.
remote: Counting objects: 100% (157/157), done.
remote: Compressing objects: 100% (129/129), done.
fatal: fetch-pack: invalid index-pack output.31 MiB | 11.59 MiB/s
jeremy@Jeremys-MacBook-Pro ~ %
thanks! missed that there's a daemon
had the second issue with big limit (like 1048576)
with smaller (100000) it works
This issue continues to occur since macOS Sonoma 14.1 as changing ulimit without disabling SIP seems to have no effect anymore.
Command output is below:
$ sl clone ssh://[email protected]/ad-astra-hs/friendsim/build-system temp
remote: Enumerating objects: 487, done.
remote: Counting objects: 100% (59/59), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 487 (delta 25), reused 0 (delta 0), pack-reused 428
Receiving objects: 100% (487/487), 70.37 MiB | 14.31 MiB/s, done.
Resolving deltas: 100% (110/110), done.
From ssh://gitlab.com/ad-astra-hs/friendsim/build-system
* [new ref] 9ad985fdc585437a8bd7f1d324a1e3ffdca144d4 -> remote/main
pulling submodule fansim-engine
pulling submodule custom_volumes/tohbra
pulling submodule custom_volumes/mixiek
pulling submodule skins/ad-astra
warning: failed to inspect working copy parent
** Sapling SCM (version 0.2.20240116) has crashed:
Traceback (most recent call last):
File "static:sapling", line 86, in run
dispatch.run(args, fin, fout, ferr, config)
File "static:sapling.dispatch", line 160, in run
status = (dispatch(req) or 0) & 255
^^^^^^^^^^^^^
File "static:sapling.dispatch", line 375, in dispatch
ret = _runcatch(req)
^^^^^^^^^^^^^^
File "static:sapling.dispatch", line 559, in _runcatch
return _callcatch(ui, req, _runcatchfunc)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.dispatch", line 568, in _callcatch
return scmutil.callcatch(ui, req, func)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.scmutil", line 153, in callcatch
return func()
^^^^^^
File "static:sapling.dispatch", line 546, in _runcatchfunc
return _dispatch(req)
^^^^^^^^^^^^^^
File "static:sapling.dispatch", line 1076, in _dispatch
ret = runcommand(
^^^^^^^^^^^
File "static:sapling.ext.undo", line 131, in _runcommandwrapper
return orig(lui, repo, cmd, fullargs, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.ext.undo", line 120, in origunsetpid
return _orig(*args, **opts)
^^^^^^^^^^^^^^^^^^^^
File "static:sapling.ext.journal", line 80, in runcommand
return orig(lui, repo, cmd, fullargs, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.ext.sparse", line 497, in _tracktelemetry
res = runcommand(lui, repo, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.ext.copytrace", line 191, in _runcommand
return orig(lui, repo, cmd, fullargs, ui, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.dispatch", line 763, in runcommand
ret = _runcommand(ui, options, cmd, d)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.dispatch", line 1117, in _runcommand
return cmdfunc()
^^^^^^^^^
File "static:sapling.dispatch", line 1075, in <lambda>
d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.util", line 1398, in check
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.util", line 1398, in check
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.ext.sparse", line 582, in _clonesparsecmd
return orig(ui, repo, *args, **opts)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.util", line 1398, in check
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.commands", line 1629, in clone
r = git.clone(ui, giturl, dest, update)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.git", line 184, in clone
hg.updatetotally(repo.ui, repo, node, None)
File "static:sapling.hg", line 980, in updatetotally
hasunresolved = _update(repo, checkout, updatecheck=updatecheck)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.hg", line 908, in update
stats = updaterepo(repo, node, False, updatecheck=updatecheck)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.ext.remotenames", line 327, in exupdate
res = orig(repo, *args, **opts)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.hg", line 894, in updaterepo
return mergemod.goto(
^^^^^^^^^^^^^^
File "static:sapling.ext.sparse", line 418, in _goto
results = orig(repo, node, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.merge", line 2003, in goto
git.submodulecheckout(p2, force=force)
File "static:sapling.git", line 728, in submodulecheckout
submod.checkout(node, force=force)
File "static:sapling.git", line 883, in checkout
hg.updaterepo(repo, node, overwrite=force)
File "static:sapling.ext.remotenames", line 327, in exupdate
res = orig(repo, *args, **opts)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.hg", line 894, in updaterepo
return mergemod.goto(
^^^^^^^^^^^^^^
File "static:sapling.ext.sparse", line 418, in _goto
results = orig(repo, node, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.merge", line 1957, in goto
isclonecheckout = repo["."].node() == nullid
~~~~^^^^^
File "static:sapling.localrepo", line 1455, in __getitem__
return context.changectx(self, changeid)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.context", line 496, in __init__
self._node = repo.names.singlenode(repo, changeid)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.namespaces", line 231, in singlenode
n = v.namemap(repo, name)
^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.namespaces", line 40, in <lambda>
bmknamemap = lambda repo, name: tolist(repo._bookmarks.get(name))
^^^^^^^^^^^^^^^
File "static:sapling.scmutil", line 1289, in __get__
obj.__dict__[self.name] = self.func(obj)
^^^^^^^^^^^^^^
File "static:sapling.localrepo", line 1313, in _bookmarks
return bookmarks.bmstore(self)
^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.bookmarks", line 81, in __init__
nm = repo.changelog.nodemap
^^^^^^^^^^^^^^
File "static:sapling.localrepo", line 96, in __get__
return super(_basefilecache, self).__get__(repo, type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "static:sapling.scmutil", line 1210, in __get__
entry.obj = self.func(obj)
^^^^^^^^^^^^^^
File "static:sapling.localrepo", line 1337, in changelog
cl = _openchangelog(self)
^^^^^^^^^^^^^^^^^^^^
File "static:sapling.localrepo", line 3519, in _openchangelog
inner = repo._rsrepo.changelog()
^^^^^^^^^^^^^^^^^^^^^^^^
error.UncategorizedNativeError: When constructing alloc::boxed::Box<dyn commits_trait::DagCommits + core::marker::Send> from dyn storemodel::StoreInfo, "10-git-commits" reported error
Caused by:
0: listing git references
1: failed to open directory '/Users/michal/Projects/temp/.sl/store/gitmodules/fb024560ddbce75570176ec9206a1a7502d65a60/.sl/store/git/refs/': Too many open files; class=Os (2); code=NotFound (-3)
