sapling
sapling copied to clipboard
facebook
Support for self-hosted GHE
I did some initial testing with a selfhosted Github Enterprise and it seems both sl pr and sl ghstack both fail to see this as a Github repo.
sl pr fails with a not a Git repo message (also should this be not a Github repo?`
sl pr
abort: not a Git repo
Seems like the error is coming from here
https://github.com/facebook/sapling/blob/c410b866312781cad42f80b1ecdd846bfae360b3/eden/scm/edenscm/ext/github/submit.py#L23-L28
Which calls
https://github.com/facebook/sapling/blob/c410b866312781cad42f80b1ecdd846bfae360b3/eden/scm/edenscm/ext/github/github_repo_util.py#L15-L23
Seems like it's hard-coded to just github.com .
sl ghstack has a similar failure and produced a stack trace saying it cannot find the repo owner / name:
$ sl ghstack
** Sapling SCM (version 20221115.080554.34470671) has crashed:
Traceback (most recent call last):
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/__init__.py", line 106, in run
dispatch.run(args, fin, fout, ferr, config)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/dispatch.py", line 161, in run
status = (dispatch(req) or 0) & 255
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/dispatch.py", line 503, in dispatch
ret = _runcatch(req)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/dispatch.py", line 712, in _runcatch
return _callcatch(ui, req, _runcatchfunc)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/dispatch.py", line 721, in _callcatch
return scmutil.callcatch(ui, req, func)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/scmutil.py", line 147, in callcatch
return func()
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/dispatch.py", line 683, in _runcatchfunc
return _dispatch(req)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/dispatch.py", line 1218, in _dispatch
ret = runcommand(
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/ext/undo.py", line 148, in _runcommandwrapper
result = orig(lui, repo, cmd, fullargs, *args)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/ext/journal.py", line 80, in runcommand
return orig(lui, repo, cmd, fullargs, *args)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/ext/sparse.py", line 504, in _tracktelemetry
res = runcommand(lui, repo, *args)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/ext/copytrace.py", line 173, in _runcommand
return orig(lui, repo, cmd, fullargs, ui, *args, **kwargs)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/dispatch.py", line 913, in runcommand
ret = _runcommand(ui, options, cmd, d)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/dispatch.py", line 1259, in _runcommand
return cmdfunc()
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/dispatch.py", line 1217, in <lambda>
d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/util.py", line 1285, in check
return func(*args, **kwargs)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/ext/ghstack/__init__.py", line 50, in ghstack_command
return submit_cmd(ui, repo, *args, **opts)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/ext/ghstack/__init__.py", line 103, in submit_cmd
ghstack.submit.main(
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ghstack/submit.py", line 136, in main
repo = ghstack.github_utils.get_github_repo_info(
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ghstack/github_utils.py", line 65, in get_github_repo_info
name_with_owner = get_github_repo_name_with_owner(
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ghstack/github_utils.py", line 41, in get_github_repo_name_with_owner
raise RuntimeError(
RuntimeError: Couldn't determine repo owner and name from url: https://github.notfacebook.com/brandonbennett/test.git
@nemith You mentioned "self-hosted," so is this something I can set up for free? I think my main challenge is a lack of a way to test this right now.
I have never tried but looks like there is a 45-day trial https://docs.github.com/en/[email protected]/get-started/signing-up-for-github/setting-up-a-trial-of-github-enterprise-server
https://enterprise.github.com/releases/3.7.0/download
I don't really know the full implications of this but got sl clone to work locally with GHE instance like this. I have not tried any other command yet 😅
def is_github_repo(repo) -> bool:
"""Create or update GitHub pull requests."""
return git.isgitpeer(repo)
# if not git.isgitpeer(repo):
# return False
# try:
# return repo.ui.paths.get("default", "default-push").url.host == "github.com"
# except AttributeError: # ex. paths.default is not set
# return False
I just got set up with access to a GHE instance, so I'll do my best to start fixing these things!
I noticed that the web interface for smartlog also doesn't work here. Will need to pipe the Git host through to https://github.com/facebook/sapling/blob/main/addons/isl-server/src/github/queryGraphQL.ts#L39 and the like. gh takes a --hostname arg for all it's commands I believe...
Yep, we have some diffs in the works for both! Haven't started on ReviewStack yet, though...
ISL fix just landed:
https://github.com/facebook/sapling/commit/289d16b494278f7ed05c5ba86c6f064c00907a77
Still waiting for our internal machinery to verify the CLI fixes.
Here's the fix for the CLI:
https://github.com/facebook/sapling/commit/eb41bc955deddcded547f91950b4eaba8bd38788
Builds for the next version are available with these fixes!
https://github.com/facebook/sapling/releases/tag/0.1.20221118-210929-cfbb68aa
If any of you who have been following this issue would be willing to try the new version (now with a semver-compliant version number!), your feedback would help a lot. I'm eager to promote this build to Latest!
Tested out Sapling 0.1.20221118-210929-cfbb68aa and I am now able to upload pr's to the GHE instance with the command sl pr
sl pr seems to fail for me against a GHE repo but it ~~night~~ might because I initialized a new repo and it has no branches yet?
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/ext/github/gh_submit.py", line 101, in get_repository
repository = _parse_repository_from_dict(repo, hostname=hostname, upstream=upstream)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/ext/github/gh_submit.py", line 157, in _parse_repository_from_dict
default_branch=repo_obj["defaultBranchRef"]["name"],
I will try again after creating a default branch. Though it seems like I'll have to do that outside of sl cli? 🤔
% sl bookmark main
abort: bookmark 'main' not allowed by configuration
brandon@Brandons-MacBook-Pro sapling_test % sl bookmark master
abort: bookmark 'master' not allowed by configuration
sl prseems to fail for me against a GHE repo but it night because I initialized a new repo and it has no branches yet?File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/ext/github/gh_submit.py", line 101, in get_repository repository = _parse_repository_from_dict(repo, hostname=hostname, upstream=upstream) File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/edenscm/ext/github/gh_submit.py", line 157, in _parse_repository_from_dict default_branch=repo_obj["defaultBranchRef"]["name"],I will try again after creating a default branch. Though it seems like I'll have to do that outside of sl cli? 🤔
% sl bookmark main abort: bookmark 'main' not allowed by configuration brandon@Brandons-MacBook-Pro sapling_test % sl bookmark master abort: bookmark 'master' not allowed by configuration
Yes, that's because the repo is empty. I thought I had a better error message in that case...The quickest fix is to initialize the repo with a README, .gitignore, or whatever through the GitHub UI.
@bolinfest also thanks for the quick work on this! Really awesome to see.
Can confirm sl pr works for me with https://github.com/facebook/sapling/releases/tag/0.1.20221118-210929-cfbb68aa 🎉 🕺
Let me know if this should be a new issue: sl ghstack seems to failing against a GHE repo for me.
return super().git(*full_args, **kwargs)
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ghstack/shell.py", line 282, in git
return self._maybe_rstrip(self.sh(*(("git",) + args), **kwargs))
File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ghstack/shell.py", line 218, in sh
raise RuntimeError(
RuntimeError: git --git-dir /Users/brandon/sapling_test2/.sl/store/git push ssh://git@REDACTEDURL/brandon/sapling_test2.git c83f72dfcfc472c5dd1ac1fcb2c39ddb34494612:refs/heads/gh/brandon/0/head 258923dc3363dc7273f184dd45c78a19d86394af:refs/heads/gh/brandon/0/base failed with exit code 128
Some quick googling suggests that this means I don't have the correct permissions but I do have full/write access to this repo. Just curious if you ran into this before
Let me know if this should be a new issue:
sl ghstackseems to failing against a GHE repo for me.return super().git(*full_args, **kwargs) File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ghstack/shell.py", line 282, in git return self._maybe_rstrip(self.sh(*(("git",) + args), **kwargs)) File "/opt/homebrew/Cellar/[email protected]/3.8.15/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ghstack/shell.py", line 218, in sh raise RuntimeError( RuntimeError: git --git-dir /Users/brandon/sapling_test2/.sl/store/git push ssh://git@REDACTEDURL/brandon/sapling_test2.git c83f72dfcfc472c5dd1ac1fcb2c39ddb34494612:refs/heads/gh/brandon/0/head 258923dc3363dc7273f184dd45c78a19d86394af:refs/heads/gh/brandon/0/base failed with exit code 128Some quick googling suggests that this means I don't have the correct permissions but I do have full/write access to this repo. Just curious if you ran into this before
Oh, this was kind of my user error, though perhaps the error message could be improved. My ssh key was not available at the time I initially ran sl ghstack. sl pull for instance revealed a permission denied error, which is the normal indicator that my ssh key was temporarily not available. Once I remediated that ghstack worked as expected 👍
Now I have a new issue when running sl ghstack:
** Sapling SCM (version 0.1.20221118-210929-cfbb68aa) has crashed:
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/edenscm/__init__.py", line 106, in run
dispatch.run(args, fin, fout, ferr, config)
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 161, in run
status = (dispatch(req) or 0) & 255
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 503, in dispatch
ret = _runcatch(req)
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 712, in _runcatch
return _callcatch(ui, req, _runcatchfunc)
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 721, in _callcatch
return scmutil.callcatch(ui, req, func)
File "/usr/lib/python3.10/site-packages/edenscm/scmutil.py", line 147, in callcatch
return func()
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 683, in _runcatchfunc
return _dispatch(req)
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 1218, in _dispatch
ret = runcommand(
File "/usr/lib/python3.10/site-packages/edenscm/ext/undo.py", line 146, in _runcommandwrapper
result = orig(lui, repo, cmd, fullargs, *args)
File "/usr/lib/python3.10/site-packages/edenscm/ext/journal.py", line 80, in runcommand
return orig(lui, repo, cmd, fullargs, *args)
File "/usr/lib/python3.10/site-packages/edenscm/ext/sparse.py", line 504, in _tracktelemetry
res = runcommand(lui, repo, *args)
File "/usr/lib/python3.10/site-packages/edenscm/ext/copytrace.py", line 173, in _runcommand
return orig(lui, repo, cmd, fullargs, ui, *args, **kwargs)
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 913, in runcommand
ret = _runcommand(ui, options, cmd, d)
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 1259, in _runcommand
return cmdfunc()
File "/usr/lib/python3.10/site-packages/edenscm/dispatch.py", line 1217, in <lambda>
d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
File "/usr/lib/python3.10/site-packages/edenscm/util.py", line 1285, in check
return func(*args, **kwargs)
File "/usr/lib/python3.10/site-packages/edenscm/ext/ghstack/__init__.py", line 51, in ghstack_command
return submit_cmd(ui, repo, *args, **opts)
File "/usr/lib/python3.10/site-packages/edenscm/ext/ghstack/__init__.py", line 103, in submit_cmd
conf, sh, github = _create_ghstack_context(ui, repo)
File "/usr/lib/python3.10/site-packages/edenscm/ext/ghstack/__init__.py", line 226, in _create_ghstack_context
github_username = github.graphql(
File "/usr/lib/python3.10/site-packages/ghstack/github_cli_endpoint.py", line 46, in graphql
result = loop.run_until_complete(make_request(params, hostname=self.hostname))
File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/usr/lib/python3.10/site-packages/ghstack/github_gh_cli.py", line 59, in make_request
assert response is not None
AssertionError
Thoughts? I've confirmed I'm logged in with gh auth status --hostname [my GHE hostname]
@skevy I uncovered an issue recently (but haven't created an issue for it) where it appears that if you don't have user.name and user.email set in .gitconfig, there are cases where ghstack does not work. This has to do with how we use Git under the hood. It's on my short list of things to fix, but note it is not a GHE issue.
@bolinfest i have both of these set. I did some further debugging yesterday. sl pr fails too. I think the issue is actually the gh version that I'm using (I'm using the latest). Calling gh api does not output raw json it seems like (even when not in a TTY)...it outputs JSON with ANSI escape sequences inside of it. My guess it that this is breaking the JSON parsing.
@skevy Hmm, I haven't heard of the ANSI escape sequence issue before. Do you have anything else installed on your system that could contribute to that? Also, how are you testing it? For example, if you did the Windows equivalent of:
gh api repos/facebook/sapling/issues > /tmp/output.json
are you saying that /tmp/output.json contains ASCII escape codes?
Can you try adding --debug with your sl ghstack call to see if that reveals anything else?
@skevy looking at https://cli.github.com/manual/gh_config, what is the output of gh config list on your host? Perhaps pager is set to something that is affecting the output?
@bolinfest this is a good guess and is definitely pointing in the right direction. I tried this on a clean linux machine and it works...so it's something to do with my shell setup that is making gh behave differently. I will spelunk and figure it out and report back to see if there's anything we should be doing here to fix it.
Sure enough. I was setting CLICOLOR_FORCE=1 in some random zsh script somewhere.
Overall, I think the takeaway I have here for Sapling is that the error message was inscrutable and didn't provide any useful information as to what was going wrong. I also tried going and putting print statements in the Python code identified in the stack trace, but that didn't work because the python seems to be byte-compiled (and apparently clearing the __pycache__ wasn't enough...it's been a while since I've worked in Python, so I'm probably missing something obvious here). And then getting Sapling to actually build and be installed properly was tough due to Python craziness in my environment haha 😀. So maybe if there was some more logging or something as to what went wrong vs a virtually un-debuggable stacktrace, that would be cool. Of course, I know that's easier said than done...so take this just as general feedback :)
Anyway, things are working now! Thanks for fixing the GHE support so quickly!
I also tried going and putting print statements in the Python code identified in the stack trace
I also tried adding print statements for a different issue I was debugging was confused about those statements not showing up. I got some tips on Sapling's Discord about how to get messages to show up in the TUI (terminal UI) though I've not been able to get it to work yet:
https://discord.com/channels/1042527964224557107/1042527965256364157/1045200362526736434
tl:dr Many methods in Sapling have a ui object as a parameter and apparently you can do ui.write to get print/log messages to show up. Or tx.io((.write in Rust code.
Sure enough. I was setting CLICOLOR_FORCE=1 in some random zsh script somewhere.
I just committed 7c0fafa26cae4319f5a560dbabdea2652410f976, which will avoid this issue going forward. We plan to make a new release today that will include this.
Overall, I think the takeaway I have here for Sapling is that the error message was inscrutable and didn't provide any useful information as to what was going wrong.
Yes, but in this case, the thing that was going wrong was so far out of the realm of things we would expect to go wrong (gh api not returning valid JSON even though the exit code was 0) that it's not completely surprising to me that we didn't have any way to surface a more appropriate error. For reference, here is the code:
https://github.com/facebook/sapling/blob/414060b3679c7132604769a9156bf7ec828701f9/eden/scm/ghstack/github_gh_cli.py#L61-L80
So to be fair, we check for quite a few things, but not the thing you happened to do because we didn't realize it was possible...