hf-hub icon indicating copy to clipboard operation
hf-hub copied to clipboard

feat/reuse blobs

Open benedikt-schaber opened this issue 1 year ago • 10 comments

Current behavior

When the file/symlink is not found in snapshots the blob is downloaded regardless of whether or not it is already present.

New behavior

When the file/symlink is not found in snapshots the etag is used to check for the corresponding blob. It is used if already present.

Potential Issues

  • The python library provides users with the option to force the download of the file. If this request is accepted we should likely do the same. I am however unaware of how such options should be introduced in this repository
  • Probably not really an issue, existed before this change was introduced: If symlinking generally works on Windows but fails once, we move the file and end up with invalid symlinks (if I am not mistaken?, not 100% sure because it is Windows, but from what I understand of CreateSymbolicLinkW and std::os::windows::fs::symlink_file).

benedikt-schaber avatar Sep 07 '23 16:09 benedikt-schaber