Git bridge?

[tobiasBora]

Hello,

First, thanks a lot for this awesome project. I'm looking for a (really) open-source equivalent of Overleaf (practical to write LaTeX with other people in real time) that I could host by myself, and this project looks very powerful and promising. The thing I'm the most interested in is the git bridge, in order to be able to sync some Cocalc files with a git repository that I can pull/push from my own laptop (some people I'm working with does not know how to use git so they would use the online interface, but I don't like to use online editors). I saw that git is installed by default, but I don't know what is the best way to automatically sync the tex files with a git repository that I could access from my computer (again, I would like to make sure that no co-author need to use git, and if possible I'd like to be able to do everything from my computer without opening my browser).

So what is the proper way to create this "git bridge"? I could imagine to manually create a git project in it, ssh into it, and write hooks to automatically sync when I want to merge stuff, but I'm not 100% sure it's possible, and I'm wondering if there is a better way.

Thank you, and keep the great job!

PS: do you have a way to make donation if I don't want to buy a full plan?

[williamstein]

We don't have a way to make a donation.

Please click the "Help" button, and we'll give you a free trial that includes network access, so you can at least fully test using git from cocalc.

Some git sync options are:

• open a terminal and run git pull; git push from the command line
• use git in the little mini terminal
• create a small Jupyter notebook with the bash kernel, and use git that way
• create a Jupyter notebook with the python3 kernel and a "sync with git" button (using ipywidgets)
• I just write some bare minimal useful git sync functionality and add it to cocalc. What precisely would you need?

-- William

[tobiasBora]

First, thank you for the answer and sorry for the time I spent to answer myself.

So what I'd like to achieve is the following: connected users wouldn't need to run any git command, and they just use the online text editor, and users that prefers git (after the setup) would just use git pull/git push, and the process would automatically sync (both way) the online and git version.

So I spent some time to try to find how to achieve that, and with as few changes as possible. I came up with this mini tutorial, let me know if you think it can be interesting to provide a way to automatically run these commands on cocalc.

I also would like to know:

1. if there is a way to force Cocalc to sync code of all users to a file (for now is seems to work pretty well automatically, but I'd like to avoid race conflicts)
2. when I modify the file .tex, it automatically updates in the code view, but the pdf does not change, even if I click on build. I need to modify the file again (with dummy modification) to compile it. Do you know why?
3. It's not directly related, but from time to time I remarked that I can't ssh anymore (it asks me a password). And after a few mn, it works again. Any idea why? Also, am I supposed to be able to ssh when the project is not started in my browser?

Thanks!

Tutorial overleaf-like git-bridge

So the first remark is the following: I think it is better to have a non-bare repository:

• the first advantage of this method is that we don't need to run git push to commit new stuff, and therefore if we change the push hook it will not be a problem
• moreover it is possible to deploy several working trees for one repository, so it shouldn't be too much change to also be able to provide an interface that put every branch in a separate folder.

So a user that would like to use this would first run:

mkdir myproject && cd myproject # optional, you can even use ~/, but when you clone by default it will put the project in a '~' folder which is not super practical.
git init
git config user.email "cocalc"
git config user.name "cocalc"

You also need to allow users to push to working branch:

git config receive.denyCurrentBranch ignore

Then, the user can put as usual a .gitignore file to exclude all the files he does not want to follow in git. For example, .gitignore could contain some sane defaults like:

*
!*/
!*.tex
!*.jpeg
!*.jpg
!*.png
!.gitignore

(check with git status to see if you remove all unwanted files)

I'll create a tex file also to test.

Now, it's time to make sure that a git add . && commit -am 'Online change' is run everytime a user does a git pull/git clone or git push. Unfortunately, there is a server hook only, but no server hook exists on for pulling, so we need to cheat a bit. So it appears that when you run git pull/git clone, the server is asked to run the program git-upload-pack, with (at last mandatory argument) the path of the repo as argument (and similarly, for git push is runs git-receive-pack), which in turn uses stdin/stdout to communicate. So if we want to trigger git pull and git clone we can do it that way:

Write a custom git-upload-pack file in a folder, say ~/bintmp/ (don't call it simply bin, or it will also change the executable for regular ssh connection) that contains the following code (just update the myproject):

#!/usr/bin/env bash
(>&2 echo "====== Let's first commit if needed =====")
cd "$GIT_PROJECT_PATH"
(git add . && git commit -am "Online change") 1>&2
(>&2 echo "====== Now let's do the clone/pull =====")
cd ~/
/usr/bin/git-upload-pack "$@"

and make sure to make it executable with:

chmod +x ~/bintmp/git-upload-pack

Now, to make sure this script is run in place of the real git-upload-pack stuff, it appears that it is possible to change the environment on ssh access (provided that the server has setup PermitUserEnvironment in the sshd (disabled by default, but enabled on Cocalc! See below for solution with PermitUserEnvironment), which is not dangerous if you already give full shell access to your users). There are two possibilities to change the environment

• easiest way is to write in the .ssh/environment the lines:

PATH=~/bintmp/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
GIT_PROJECT_PATH=~/myproject

• the other option is to change the .ssh/authorized_keys and configure the environment in the environment command (written before the keys), but I think that cocalc does not really encourage to modify this file so I guess the above method is prefered. Note that when you can change the authorized_keys file, you can change the environment for only one specific key.

For pushing, you also want the same thing, but create this time a file ~/bintmp/git-receive-pack and put:

#!/usr/bin/env bash
(>&2 echo "====== Let's first commit if needed =====")
cd "$GIT_PROJECT_PATH"
(git add . && git commit -am "Online change") 1>&2
(>&2 echo "====== Now let's do the push =====")
cd ~/
/usr/bin/git-receive-pack "$@"
(>&2 echo "====== Push finished, let's reset on server =====")
cd "$GIT_PROJECT_PATH"
git reset --hard HEAD 1>&2

and we make it executable as well:

chmod +x ~/bintmp/git-receive-pack

Enjoy!

PS: Note that if the server does not have PermitUserEnvironment (so does not concern Cocalc), it is possible to use the command command in the .ssh/authorized_keys file to overwrite the real command (that's the method used by gitolite). It's a bit more complicated, because you need to run yourself the command of the user stored in SSH_ORIGINAL_COMMAND afterwards, and you need to run bash yourself if SSH_ORIGINAL_COMMAND is not set, but it's definitely doable if you can modify the .ssh/authorized_keys, and this gives precise access on a per-key basis.

[slel]

Related: I heard HackMD has a way to synchronize markdown files so that users can collaboratively edit a markdown file on hackmd.io and use a button to sync with a version of the same file hosted in a git repo on GitHub or GitLab.

[jorsn]

@tobiasBora: Thanks a lot for this tutorial. However, for me the prefix ~ didn't work properly in the environment variables. I replaced it by /home/user. Also, .ssh/environment will be overwritten from time to time (nowadays). So, one should set env vars in the project settings web interface. However, one might not even need them, if one just uses /home/user/bin (see below) and a hardcoded git project dir.

Write a custom git-upload-pack file in a folder, say ~/bintmp/ (don't call it simply bin, or it will also change the executable for regular ssh connection)

-- @tobiasBora

Why this? If ~/bintmp is in the path before /usr/bin, it has exactly the same effect as using ~/bin, since ~/.ssh/environment does not distinguish between different ssh logins. Also, when setting the PATH in the project settings, it takes effect for all processes started by the user in the container, as far as I understand.

[jorsn]

I created a script that does the setup: https://gist.github.com/jorsn/398ba31ac6fbfcf1c25eb6e0e07a3773

[williamstein]

This is great, but I don't think it needs to be an open issue.

Incidentally, another way to have a graphical Git client is via VS Code, for what it is worth, which is usable with cocalc via 1 click (from the file explorer).