jupyterlab-git
jupyterlab-git copied to clipboard
jupyterlab
"Unable to get lock on the directory" as several git processes continue running
Tried fixing this for quite some time, but couldn't find a solution by searching. Maybe this is known already, but couldn't find an issue, except #873
Description
After starting jupyterlab and entering any repository the git sidebar shows as expected. As soon as I run any git action (refresh, pull, push, checkouts) or the extension does the refresh by itself the pull / push buttons get greyed out and I get message "Unable to get lock on the directory", when I try to pull/push. No more git actions possible.
I checked if .git/index.lock got stuck, but it doesn't exist. I checked processes. No git processes before git processes, any action in interface spins off several git processes, which continue to run, unless killed manually:
PID TTY TIME CMD
23689 pts/8 00:00:58 screen
48703 pts/13 00:00:37 screen
94134 pts/1 00:00:00 ps
102221 pts/18 14:53:05 jupyter-noteboo
124261 pts/0 00:00:12 jupyter-lab
127234 pts/0 00:00:00 git
127246 pts/0 00:00:00 git
127253 pts/0 00:00:00 git
127255 pts/0 00:00:00 git-remote-http
When I run killall git, the pull/push buttons are black again, but only until next refresh or other action.
Further I checked permission rights for ./.git directory, which look fine:
drwxrwxr-x 8 christian christian 4096 Jul 12 14:40 .git
It looks like too many git processes are launched and block each other. In previous version's normal behaviour only one process runs and stops after finishing.
- git is working in terminal (tested separately, but not parallel), credentials stored with helper.
Reproduce
- Go to jupyterlab and enter repository in file browser
- Click on git extension sidebar
- Either wait for automatic refresh or do any git action (pull/push/checkout branch)
- Receive error message "Unable to get lock on the directory" I reproduced it with other linux machine as well.
Expected behavior
- git processes started by jupyterlab-git should finish after completion.
- Also tried running the extention in version 0.22.1 in jupyterlab 2.2.8 and it works just fine (of course, this was tested not in parallel). In this setup the git processes quit after finishing.
Command Line Output Error (jupyter lab --debug)
Context
- Python package version:
Name: jupyterlab-git
Version: 0.30.1
- Extension version:
JupyterLab v3.0.16
/home/christian/anaconda3/envs/jl3/share/jupyter/labextensions
jupyterlab-execute-time v2.0.4 enabled OK (python, jupyterlab_execute_time)
nbdime-jupyterlab v2.1.0 enabled OK
@jupyter-widgets/jupyterlab-manager v3.0.0 enabled OK (python, jupyterlab_widgets)
@ryantam626/jupyterlab_code_formatter v1.4.10 enabled OK (python, jupyterlab-code-formatter)
@jupyterlab/git v0.30.1 enabled OK (python, jupyterlab-git)
- Git version:
git version 2.30.0 - Operating System and its version:
Ubuntu 18.04.5 LTS
Environment info
## Complete Environment : Please see environment.yml file as txt attached (I counldn't upload yml). [environment_jl3test.yml.txt](https://github.com/jupyterlab/jupyterlab-git/files/6802070/environment_jl3test.yml.txt) ``` $ conda list # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge anyio 3.2.1 py39hf3d152e_0 conda-forge appdirs 1.4.4 pypi_0 pypi argon2-cffi 20.1.0 py39h3811e60_2 conda-forge async_generator 1.10 py_0 conda-forge attrs 21.2.0 pyhd8ed1ab_0 conda-forge autopep8 1.5.7 pypi_0 pypi babel 2.9.1 pyh44b312d_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge black 21.6b0 pypi_0 pypi bleach 3.3.0 pyh44b312d_0 conda-forge brotlipy 0.7.0 py39h3811e60_1001 conda-forge ca-certificates 2021.7.5 h06a4308_1 certifi 2021.5.30 py39h06a4308_0 cffi 1.14.5 py39he32792d_0 conda-forge chardet 4.0.0 py39hf3d152e_1 conda-forge click 8.0.1 pypi_0 pypi colorama 0.4.4 pyh9f0ad1d_0 conda-forge cryptography 3.4.7 py39hbca0aa6_0 conda-forge decorator 5.0.9 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge entrypoints 0.3 pyhd8ed1ab_1003 conda-forge gitdb 4.0.7 pyhd8ed1ab_0 conda-forge gitpython 3.1.18 pyhd8ed1ab_0 conda-forge icu 68.1 h2531618_0 idna 2.10 pyh9f0ad1d_0 conda-forge importlib-metadata 4.6.0 py39hf3d152e_0 conda-forge ipykernel 5.5.5 py39hef51801_0 conda-forge ipython 7.25.0 py39hef51801_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge jedi 0.18.0 py39hf3d152e_2 conda-forge jinja2 3.0.1 pyhd8ed1ab_0 conda-forge json5 0.9.5 pyh9f0ad1d_0 conda-forge jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge jupyter-server-mathjax 0.2.3 pyhd8ed1ab_0 conda-forge jupyter_client 6.1.12 pyhd8ed1ab_0 conda-forge jupyter_core 4.7.1 py39hf3d152e_0 conda-forge jupyter_server 1.9.0 pyhd8ed1ab_0 conda-forge jupyterlab 3.0.16 pyhd8ed1ab_0 conda-forge jupyterlab-git 0.30.1 pyhd8ed1ab_0 conda-forge jupyterlab_code_formatter 1.4.10 pyhd8ed1ab_1 conda-forge jupyterlab_execute_time 2.0.4 pyhd8ed1ab_0 conda-forge jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge jupyterlab_server 2.6.0 pyhd8ed1ab_0 conda-forge jupyterlab_widgets 1.0.0 pyhd8ed1ab_1 conda-forge ld_impl_linux-64 2.35.1 hea4e1c9_2 conda-forge libffi 3.3 h58526e2_2 conda-forge libgcc-ng 9.3.0 h2828fa1_19 conda-forge libgomp 9.3.0 h2828fa1_19 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libstdcxx-ng 9.3.0 h6de172a_19 conda-forge libuv 1.40.0 h7b6447c_0 markupsafe 2.0.1 py39h3811e60_0 conda-forge matplotlib-inline 0.1.2 pyhd8ed1ab_2 conda-forge mistune 0.8.4 py39h3811e60_1003 conda-forge mypy-extensions 0.4.3 pypi_0 pypi nbclassic 0.3.1 pyhd8ed1ab_1 conda-forge nbclient 0.5.3 pyhd8ed1ab_0 conda-forge nbconvert 6.1.0 py39hf3d152e_0 conda-forge nbdime 3.1.0 pyhd8ed1ab_0 conda-forge nbformat 5.1.3 pyhd8ed1ab_0 conda-forge ncurses 6.2 h58526e2_4 conda-forge nest-asyncio 1.5.1 pyhd8ed1ab_0 conda-forge nodejs 14.8.0 hda19d22_0 notebook 6.4.0 pyha770c72_0 conda-forge openssl 1.1.1k h27cfd23_0 packaging 20.9 pyh44b312d_0 conda-forge pandoc 2.14.0.3 h7f98852_0 conda-forge pandocfilters 1.4.2 py_1 conda-forge parso 0.8.2 pyhd8ed1ab_0 conda-forge pathspec 0.8.1 pypi_0 pypi pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pip 21.1.3 pyhd8ed1ab_0 conda-forge prometheus_client 0.11.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.19 pyha770c72_0 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pycodestyle 2.7.0 pypi_0 pypi pycparser 2.20 pyh9f0ad1d_2 conda-forge pygments 2.9.0 pyhd8ed1ab_0 conda-forge pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyrsistent 0.17.3 py39h3811e60_2 conda-forge pysocks 1.7.1 py39hf3d152e_3 conda-forge python 3.9.5 h49503c6_0_cpython conda-forge python-dateutil 2.8.1 py_0 conda-forge python_abi 3.9 2_cp39 conda-forge pytz 2021.1 pyhd8ed1ab_0 conda-forge pyzmq 22.1.0 py39h37b5a0c_0 conda-forge readline 8.1 h46c0cb4_0 conda-forge regex 2021.4.4 pypi_0 pypi requests 2.25.1 pyhd3deb0d_0 conda-forge requests-unixsocket 0.2.0 py_0 conda-forge send2trash 1.7.1 pyhd8ed1ab_0 conda-forge setuptools 49.6.0 py39hf3d152e_3 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge smmap 3.0.5 pyh44b312d_0 conda-forge sniffio 1.2.0 py39hf3d152e_1 conda-forge sqlite 3.36.0 h9cd32fc_0 conda-forge terminado 0.10.1 py39hf3d152e_0 conda-forge testpath 0.5.0 pyhd8ed1ab_0 conda-forge tk 8.6.10 h21135ba_1 conda-forge toml 0.10.2 pypi_0 pypi tornado 6.1 py39h3811e60_1 conda-forge traitlets 5.0.5 py_0 conda-forge typing_extensions 3.10.0.0 pyha770c72_0 conda-forge tzdata 2021a he74cb21_0 conda-forge urllib3 1.26.6 pyhd8ed1ab_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webencodings 0.5.1 py_1 conda-forge websocket-client 0.57.0 py39hf3d152e_4 conda-forge wheel 0.36.2 pyhd3deb0d_0 conda-forge xz 5.2.5 h516909a_1 conda-forge zeromq 4.3.4 h9c3ff4c_0 conda-forge zipp 3.4.1 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h516909a_1010 conda-forge ```
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:
![welcome[bot] avatar](https://avatars.githubusercontent.com/in/4141?v=4 "Published by a pub.dev verified publisher"){kind=small}
Thanks @chrisguettler for the detailed report. We definitely need to figure out what is going out as you seem to hit a lock situation.
Some questions:
- when listing the process could you run
ps aux? so we have the full executed command, not only the application name. - does your system as complex remote access like entreprise proxy?
- in the extension settings, you can increase the delay between two refresh status call. Could you try to increase it by a huge amount to see if this is the command that triggers the lock?
- is the credentials helper asking for some validation or are the credentials stored once and for all - so that you don't need to set them when starting JupyterLab?
Hi @fcollonval, sorry for not having answered earlier (business & vacation).
ps aux shows:
christi+ 72915 0.0 0.0 18932 3964 pts/8 S+ 07:03 0:00 git fetch --all --prune
christi+ 72916 0.0 0.0 19196 3952 pts/8 S+ 07:03 0:00 /usr/lib/git-core/git fetch --append --prune origin
christi+ 72917 0.0 0.0 196472 22016 pts/8 S+ 07:03 0:00 /usr/lib/git-core/git-remote-https origin https://...
The envrionment runs on a remote enterprise setup, but no proxy involved, just SSH connection.
When I extend the refresh intervall the automatic refresh does not cause the issue anymore.
I didn't store credentials in with the helper. I noticed, as soon as I kill all git processes, it's prompting for credentials in the terminal, where jupyterlab is running.
[E 2021-08-25 11:06:48.376 ServerApp] 500 POST /git/remote/fetch?1629882235718 (10.23.10.13) 172617.79ms referer=http://[machine address]:8882/lab
Password for [repo]:
But it did not prompt for it in the browser. Website is shown as "not secure" in Chrome. So I allowed pop-ups and unsecure contents for the address. It now did prompt for username/password and doesn't block anymore. so far so good. Storing credentials (local) in helper also helps, but only temporarily.
If I set back refresh intervall to 3000 and restart JL3 the issue continues unless I kill all git processes again and enter the credentials in the JL-Terminal, again no prompt in the browser. If I leave the refresh intervall incredibly high, I have no issues again, manual refresh, pull, push works fine. It looks like the refresh caused by the intervall, doesn't prompt for credentials.
By the way, if I set the intervall to higher than 2000000000 or so the git panel disappears, when reloading the page.
I had the same problem, two options;
- ssh keys; you need to add the key to the manager!! see https://stackoverflow.com/questions/1595848/configuring-git-over-ssh-to-login-once
- https login use something like git config --global credential.helper cache
Is this issue resolved? Are there any suggested workarounds?
What are exact steps that I should follow to get past the following error, while trying to pull.
Error: Unable to get the lock on the directory
Can someone help me on this. Thank you.
I get the problem if l try to use jupyter lab with git on another computer than my own. The workaround is set up git with ssh keys