PermissionError: [WinError 5]

[Magniveo]

I have an error in Windows when running litgpt download --repo_id microsoft/phi-2 The error may be in os.remove(path)

[Magniveo]

try:
     os.remove(safetensor_path)
     print(f"Removed {safetensor_path}")
     os.rename(bin_path, safetensor_path)
     print(f"Renamed {bin_path} to {safetensor_path}")
except:
           continue
            pass

[rasbt]

Thanks for reporting! Unfortunately, I don't have a windows machine to test it, but we need to investigate ... Maybe @carmocca can help here as I think he has a non-mac computer.

[carmocca]

I use Linux. @Magniveo What's the error you get exactly? Are you able to reproduce the issue in a script that only runs those lines? Do you have proper permissions in that directory?

[Magniveo]

The error PermissionError: [WinError 5] persists, even after attempting to solve it with administrative privileges.

[carmocca]

Does

import shutil
shutil.rmtree(safetensor_path)

work?

[Magniveo]

Does

import shutil
shutil.rmtree(safetensor_path)

work?

Nope, there's another error File "C:\Python312\Lib\shutil.py", line 621, in _rmtree_unsafe with os.scandir(path) as scandir_it: NotADirectoryError: [WinError 267]

[carmocca]

My bad. I thought rmtree worked for files

[rasbt]

I use Linux.

Ah my bad, I thought you might have a dual boot or WSL setup. Now I know :)

[Hungryarie]

I can confirm this error (windows 10 machine without WSL). PermissionError: [WinError 5] Access is denied: 'checkpoints\\microsoft\\phi-2\\model-00001-of-00002.safetensors' the error happens for other LLM's as well.

commenting out line 104 of download.py does the trick.

whole error: command run: litgpt download microsoft/phi-2

PS C:\Data\code\litGPT> litgpt download microsoft/phi-2
repo_id: microsoft/phi-2
Setting HF_HUB_ENABLE_HF_TRANSFER=1
C:\Data\apps\v_envs\py310_gpt\lib\site-packages\huggingface_hub\file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
C:\Data\apps\v_envs\py310_gpt\lib\site-packages\huggingface_hub\file_download.py:1194: UserWarning: `local_dir_use_symlinks` parameter is deprecated and will be ignored. The process to download files to a local folder has been updated and do not rely on symlinks anymore. You only need to pass a destination folder as`local_dir`.
For more details, check out https://huggingface.co/docs/huggingface_hub/main/en/guides/download#download-files-to-local-folder.
  warnings.warn(
Converting .safetensor files to PyTorch binaries (.bin)
checkpoints\microsoft\phi-2\model-00001-of-00002.safetensors --> checkpoints\microsoft\phi-2\model-00001-of-00002.bin
Traceback (most recent call last):
  File "C:\Program Files\Python-3102\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python-3102\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Data\apps\v_envs\py310_gpt\Scripts\litgpt.exe\__main__.py", line 7, in <module>
    sys.exit(main())
  File "C:\Data\apps\v_envs\py310_gpt\lib\site-packages\litgpt\__main__.py", line 57, in main
    CLI(parser_data)
  File "C:\Data\apps\v_envs\py310_gpt\lib\site-packages\jsonargparse\_cli.py", line 119, in CLI
    return _run_component(component, init.get(subcommand))
  File "C:\Data\apps\v_envs\py310_gpt\lib\site-packages\jsonargparse\_cli.py", line 196, in _run_component
    return component(**cfg)
  File "C:\Data\apps\v_envs\py310_gpt\lib\site-packages\litgpt\scripts\download.py", line 104, in download_from_hub
    os.remove(safetensor_path)
PermissionError: [WinError 5] Access is denied: 'checkpoints\\microsoft\\phi-2\\model-00001-of-00002.safetensors'

[Andrei-Aksionov]

Windows is odd. What makes it a bit more "spicy" is that, apparently, none of the maintainers have Windows machine. Debugging it via CI is sub-optimal, to say at least.

So far I've tried manually change permission to the least restrictive (777) but it didn't help.

I guess the best option for right now is to catch this error and print a message saying that a user has to delete .safetensor file manually to save some disk space. What do you think @rasbt?

[Hungryarie]

Windows is odd.

Yes I know, but my current company machine only has windows (without WSL activated unfortunately) your suggestion to print an message sounds helpful for the user to me. Thanks!

[rasbt]

Thanks for investigating @Hungryarie, this level of information is really appreciated!

@Andrei-Aksionov I agree, let's catch this and print a message (done via #1507 )

PS: I can't really test it, but I wonder if

file_path = Path('your_file.txt')
file_path.unlink()

could work (not sure how it differs under the hood, whether it just calls os.remove or does something additionally)

[Andrei-Aksionov]

I have already tried it :) And it didn't work.

~I'll create a PR a bit later~ Sebastian was faster.