Invalid permisions of /var/tmp/dvc/repo

[petrchmelar]

Edited by @shcheklein : See the latest comments, need some research to support the latest platformdirs (?)

Bug Report

dvc ls: creates /var/tmp/dvc with incorrect permissions

Description

Since https://github.com/iterative/dvc/releases/tag/2.48.0 the dvc is using /var/tmp/dvc as a "git cache" for unix based operating systems. When the /var/tmp/dvc does not exist the dvc creates site cache folder using os.makedirs with default 777 permissions, but these permissions are masked by UMASK (https://stackoverflow.com/a/5231994). This can cause an unexpected behavior of dvc ls (an other commands) for multi-user environment where other users can not write to originally created site-cache folder

https://github.com/iterative/dvc/blob/612f3039e05b609d02d96c063763f8c093c2779d/dvc/repo/init.py#L212

Reproduce

1. remove site cache dir rm -r -f /var/tmp/dvc
2. run dvc ls dvc ls [email protected]:iterative/dvc-test.git
3. check permissions ls -al /var/tmp/dvc or run 2) as a different user

Expected

I believe that the site cache dir should be created with permissions that allows other users to list dvc repositories.

Environment information

Centos8

╰─❯ dvc doctor
DVC version: 2.48.0 (pip)
-------------------------
Platform: Python 3.8.13 on Linux-4.18.0-408.el8.x86_64-x86_64-with-glibc2.2.5
Subprojects:
	dvc_data = 0.43.0
	dvc_objects = 0.21.1
	dvc_render = 0.2.0
	dvc_task = 0.2.0
	scmrepo = 0.1.15
Supports:
	http (aiohttp = 3.8.1, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2023.1.0, boto3 = 1.24.59)
Cache types: reflink, hardlink, symlink
Cache directory: xfs on /dev/md0
Caches: local
Remotes: s3
Workspace directory: xfs on /dev/md0
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/8e5a699173beb5035173f8252c5bcbec

[petrchmelar]

Thanks a lot @efiop <3

[efiop]

@petrchmelar 2.50 should be out in a few minutes, please give it a try. You'll need to delete /var/tmp/dvc yourself one time or chmod /var/tmp/dvc and /var/tmp/dvc/repo to 0777 yourself.

Thank you for the feedback! 🙏

[petrchmelar]

I've tried to delete the /var/tm/dvc and list dvc files again and everythings seems to be working as expected. :+1:

[VK-T]

The same exceptions raised when executing dvc doctor or dvc repro and produces the error ERROR: unexpected error - [Errno 13] Permission denied: '/var/cache/dvc'. Followed the above steps to delete the files. After deletion, I am facing the same error as below:

`Traceback (most recent call last): File "/home/merit/anaconda3/envs/jato_ner/lib/python3.10/site-packages/dvc/cli/init.py", line 211, in main ret = cmd.do_run() File "/home/merit/anaconda3/envs/jato_ner/lib/python3.10/site-packages/dvc/cli/command.py", line 41, in do_run return self.run() File "/home/merit/anaconda3/envs/jato_ner/lib/python3.10/site-packages/dvc/commands/version.py", line 16, in run dvc_info = get_dvc_info() File "/home/merit/anaconda3/envs/jato_ner/lib/python3.10/site-packages/dvc/info.py", line 38, in get_dvc_info with Repo() as repo:

File "/home/merit/anaconda3/envs/jato_ner/lib/python3.10/site-packages/dvc/repo/init.py", line 220, in init os.makedirs(self.site_cache_dir, exist_ok=True) File "/home/merit/anaconda3/envs/jato_ner/lib/python3.10/site-packages/funcy/objects.py", line 25, in get res = instance.dict[self.fget.name] = self.fget(instance) File "/home/merit/anaconda3/envs/jato_ner/lib/python3.10/site-packages/dvc/repo/init.py", line 641, in site_cache_dir os.makedirs(repos_dir, mode=0o777, exist_ok=True) File "/home/merit/anaconda3/envs/jato_ner/lib/python3.10/os.py", line 215, in makedirs makedirs(head, exist_ok=exist_ok) File "/home/merit/anaconda3/envs/jato_ner/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) PermissionError: [Errno 13] Permission denied: '/var/cache/dvc'`

Configuration: DVC version: 3.37.0 (pip) Platform: Python 3.10.12 on Linux-5.4.0-124-generic-x86_64-with-glibc2.31

Please provide the suggestions to rectify the issue?

[efiop]

@VK-T Check pip check, you likely have a newer platformdirs installed that we don't support yet.

[VK-T]

@efiop Thanks for the quick reply. I'm currently using platformdirs version 3.11, even though the latest available version is 4.1.0. Given that DVC relies on platformdirs, could you advise on the optimal version to use in this context?

[efiop]

@VK-T So does pip check show any problems with it? Make sure you run in the same env that dvc is installed in.

We require platformdirs<4,>=3.1.1 per https://github.com/iterative/dvc/blob/dad464285e3f107c9e4ccc6e0b82672578037afb/pyproject.toml#L54

[VK-T]

@efiop There is no any problem with pip check. I just uninstalled and re-installed the newer version of dvc with platformdirs=3.11.1 and working fine now. Thanks

[thaiminhpv]

Got the same error today

> dvc pull
ERROR: unexpected error - [Errno 13] Permission denied: '/var/cache/dvc'

> pip list | grep dvc
dvc                          3.49.0

Since platformdirs==3.11.1 no longer exists on pypi https://pypi.org/project/platformdirs/#history , downgrade platformdirs to 3.10.0 works for me

pip install platformdirs==3.10.0

[Jack-Lewis1]

Same issue.

[WYVERN2742]

Just encountered this issue; downgrading from platformdirs-3.11.0 to platformdirs-3.10.0 fixed it for me :+1:

[mattrmd]

Same issue for DVC version 3.52.0. My error was thrown on dvc add and was as follows

ERROR: unexpected error - [Errno 13] Permission denied: '/var/cache/dvc'

Downgrading from platformdirs-3.11.0 to platformdirs-3.10.0 fixed the issue even though the former is supported according to the pyproject.toml file (see here).

[amej]

@shcheklein In line with best practices adopted by open source, generally a new issue referencing this 'closed' one is created instead of reopening a closed one. What is the practice in the 'iterative' org?

[anunayasri]

Hi. I was looking at this issue. If I understand this correctly, dvc pull throws a permission error for platformdirs-3.11.0 but not for platformdirs-3.10.0. So, what are the next steps to fix this? Doesn't this look like an issue with paltformdirs lib? If someone can help, I am happy ot take this up. cc: @shcheklein

[shcheklein]

dvc pull throws a permission error for platformdirs-3.11.0 but not for platformdirs-3.10.0

yes, it would be nice to confirm this though @anunayasri

If someone can help, I am happy ot take this up.

sounds good, let us know what kind of help you are looking for! I think the first step can be to reproduce the issue and figure out what exactly is causing it.

[anunayasri]

Sure. Let me try it out.

[anunayasri]

I am unable to reproduce this error. What I am trying to repro -

1. I am using Macos Monterey version 12.7.6
2. I don't see a directory /var/tmp/dvc which mentioned in the issue.
3. I created a dvc project called dvc-demo, which obviously has a dependency of dvc. I installed dvc in editable mode using pip install -e /path/to/local/dvc. I confirmed that dvc-demo is using the local dvc.
4. In dvc-demo, I executed dvc pull using platformdirs 3.10 and 3.11. Both worked perfectly.
5. Please note that I chanaged the platformdirs version in the dvc project.

Am I missing something here?

cc: @VK-T

[shcheklein]

@anunayasri do you have access to some virtual machine - clean Linux, to try to run it?

[anunayasri]

@shcheklein I don't have a physical machine with linux. Let me try with a docker container.

[anunayasri]

@shcheklein I tried using docker, but still unable to reproduce the error.

1. Used the image python:3.12-alpine
2. In a dvc project dvc-demo, install local clone of dvc that installs platformdir 3.11 version.
3. dvc ls https://github.com/iterative/dvc-test.git works fine with permissions in /var/tmp/dvc as 777.
4. pip uninstall platformdirs. pip install platformdirs==3.10.
5. rm -rf /var/tmp/dvc
6. Switched to a different user.
7. Ran dvc ls again. Works fine with permissions in /var/tmp/dvc as 777.

Am I missing something here?

cc: @petrchmelar @VK-T

[anunayasri]

@shcheklein Did you get a chance to go through my previous msg. I tried reproducing the error but wasn't able to. Do you think I am missing something?

[shcheklein]

hey @anunayasri ,let's close it then and wait for someone to share the exact environment where we can reproduce it.

[Red-Eyed]

I have same issue on SLURM