gitpython-developers
`git.repo.base.repo.tree()` blocks deletion of repo directory on Windows (fine in Linux)
I am checking out repositories into a temporary directory, doing various things with them including calling tree(), and then deleting the directories and the temporary directory. This works fine in Linux but not in Windows. I have created this SSCCE to show the issue. I will take some time right now to try to attempt to fix it myself but I like to document my issues first (so others know and just in case someone else sees it and has insight).
Note that I am making an effort (del_rw()) to clear read-only items.
I am testing on:
- Windows 10 Enterprise 64-bit (in VirtualBox)
- Python 3.5.2 32-bit
Output:
(venv) C:\Users\IEUser\Desktop\407>python t.py
Traceback (most recent call last):
File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 381, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\IEUser\\AppData\\Local\\Temp\\tmprbrn0gen\\tree\\.git\\objects\\pack\\pack-986f782a1c797b94e8a9fa75402bf81fbbc4d537.idx'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "t.py", line 40, in <module>
shutil.rmtree(dir, onerror=del_rw)
File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 488, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 378, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 378, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 378, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 383, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "t.py", line 19, in del_rw
os.remove(name)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\IEUser\\AppData\\Local\\Temp\\tmprbrn0gen\\tree\\.git\\objects\\pack\\pack-986f782a1c797b94e8a9fa75402bf81fbbc4d537.idx'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "t.py", line 40, in <module>
shutil.rmtree(dir, onerror=del_rw)
File "C:\Users\IEUser\Desktop\407\venv\lib\tempfile.py", line 808, in __exit__
self.cleanup()
File "C:\Users\IEUser\Desktop\407\venv\lib\tempfile.py", line 812, in cleanup
_shutil.rmtree(self.name)
File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 488, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 378, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 378, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 378, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 378, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 383, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\IEUser\Desktop\407\venv\lib\shutil.py", line 381, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\IEUser\\AppData\\Local\\Temp\\tmprbrn0gen\\tree\\.git\\objects\\pack\\pack-986f782a1c797b94e8a9fa75402bf81fbbc4d537.idx'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "t.py", line 42, in <module>
raise PermissionError('Failed with tree (GitPython {})'.format(gitpython_hash)) from e
PermissionError: Failed with tree (GitPython 5149c807ec5f396c1114851ffbd0f88d65d4c84f)
After the tree() call there are two entire Git processes that are left open... Pausing in the debugger and killing those process trees manually (via Process Explorer) does allow clean deletion of the directories.
The 'offending' instances of Git are launched per the below stack traces. I did snip the last few stack frames that were for the PyCharm debugger activities.
>>> import traceback; traceback.print_stack()
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\pydevd.py", line 1580, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\pydevd.py", line 964, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/IEUser/Desktop/407/t.py", line 40, in <module>
tree = repo.tree('master')
File "c:\users\ieuser\desktop\407\gitpython\git\repo\base.py", line 442, in tree
return self.rev_parse(text_type(rev) + "^{tree}")
File "c:\users\ieuser\desktop\407\gitpython\git\repo\fun.py", line 193, in rev_parse
obj = name_to_object(repo, rev[:start])
File "c:\users\ieuser\desktop\407\gitpython\git\repo\fun.py", line 130, in name_to_object
return Object.new_from_sha(repo, hex_to_bin(hexsha))
File "c:\users\ieuser\desktop\407\gitpython\git\objects\base.py", line 64, in new_from_sha
oinfo = repo.odb.info(sha1)
File "c:\users\ieuser\desktop\407\gitpython\git\db.py", line 37, in info
hexsha, typename, size = self._git.get_object_header(bin_to_hex(sha))
File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 930, in get_object_header
cmd = self._get_persistent_cmd("cat_file_header", "cat_file", batch_check=True)
File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 913, in _get_persistent_cmd
cmd = self._call_process(cmd_name, *args, **options)
File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 868, in _call_process
return self.execute(call, **_kwargs)
File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 586, in execute
proc = Popen(command,
and
>>> import traceback; traceback.print_stack()
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\pydevd.py", line 1580, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\pydevd.py", line 964, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/IEUser/Desktop/407/t.py", line 40, in <module>
tree = repo.tree('master')
File "c:\users\ieuser\desktop\407\gitpython\git\repo\base.py", line 442, in tree
return self.rev_parse(text_type(rev) + "^{tree}")
File "c:\users\ieuser\desktop\407\gitpython\git\repo\fun.py", line 216, in rev_parse
obj = to_commit(obj).tree
File "C:\Users\IEUser\Desktop\407\venv\lib\site-packages\gitdb\util.py", line 237, in __getattr__
self._set_cache_(attr)
File "c:\users\ieuser\desktop\407\gitpython\git\objects\commit.py", line 143, in _set_cache_
binsha, typename, self.size, stream = self.repo.odb.stream(self.binsha) # @UnusedVariable
File "c:\users\ieuser\desktop\407\gitpython\git\db.py", line 42, in stream
hexsha, typename, size, stream = self._git.stream_object_data(bin_to_hex(sha))
File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 947, in stream_object_data
cmd = self._get_persistent_cmd("cat_file_all", "cat_file", batch=True)
File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 913, in _get_persistent_cmd
cmd = self._call_process(cmd_name, *args, **options)
File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 868, in _call_process
return self.execute(call, **_kwargs)
File "c:\users\ieuser\desktop\407\gitpython\git\cmd.py", line 586, in execute
proc = Popen(command,
Does not seem to work (Still throws exception when deleting directory. I guess this just means it's not setup as a context manager):
with git.Repo.clone_from(url, dir) as repo:
with repo.tree('master') as tree:
pass
Does work (No exception when deleting directory):
repo = git.Repo.clone_from(url, dir)
tree = repo.tree('master')
del repo
del tree
So, perhaps it is not a bug and I simply need to make sure that my repo and tree get deleted?
@altendky Thanks for posting your progress here, as I do hope that others will find it useful! It's a known limitation of GitPython and its API, and best I can offer right now is to manually handle it by implementing some sort of workaround.
- OS Name: Microsoft Windows 10 Pro
- OS Version: 10.0.19045
- Python version: 3.10.9
Workaround
def main():
with git.Repo.clone_from(REPO, local_path) as repo:
# omitted for brevity
print(f"Removing temp destination directory {local_path}")
git.rmtree(local_path)
I had the same problem just with creating a repo and deleting it again. Thankfully, @gdyrrahitis solution worked for me :rocket:
