kedro-plugins icon indicating copy to clipboard operation
kedro-plugins copied to clipboard

fix(datasets): Use `put()` and `get()` instead of `copy` in `TensorFlowModelDataset`'s `_save` and `_load` methods.

Open gitgud5000 opened this issue 5 months ago • 0 comments

Description

This PR resolves the issue in #839, where saving TensorFlowModelDataset to Azure Blob Storage fails due to incorrect use of .copy() from the fsspec.filesystem interface.

Development notes

The issue occurs in the _save and _load methods on lines 147 and 172, where .copy() is used. According to the fsspec documentation, .copy() is intended for remote-to-remote transfers. Since this involves copying from a local filesystem to remote storage, .put() should be used for saving and .get() for loading.

The code has been updated to use .put() and .get() accordingly, replacing the use of .copy().

Both methods work for local-to-local and local-to-remote(& vice versa) transfers based on testing.

Checklist

  • [ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • [ ] Updated the documentation to reflect the code changes
  • [ ] Added a description of this change in the relevant RELEASE.md file
  • [ ] Added tests to cover my changes

gitgud5000 avatar Sep 22 '24 04:09 gitgud5000