luigi icon indicating copy to clipboard operation
luigi copied to clipboard

Bug in luigi.contrib.ftp.py on putting file to ftp from Windows

Open TychonautVII opened this issue 5 years ago • 3 comments

When calling the "put" operation from a windows machine using luigi.contrib.ftp.RemoteFileSystem, the code fails with a max recursion depth error.

This seems to be due the call to normpath in the beginning of both the put methods

    def _sftp_put(self, local_path, path, atomic):
        normpath = os.path.normpath(path)
        directory = os.path.dirname(normpath)

From the OS docs https://docs.python.org/3/library/os.path.html, normath "On Windows, it converts forward slashes to backward slashes." Most FTP servers use forward slashes as separators, so this messes up the remote path.

I think adding somthing like

    def _sftp_put(self, local_path, path, atomic):
        normpath = os.path.normpath(path)
        directory = os.path.dirname(normpath)
        if os.path.sep =='\\':
            directory = directory.replace('\\','//')

might just be enough to fix it

I believe that this is a dupe of a ticket @elessarelfstone put in in the past that got closed due to inactivity. https://github.com/spotify/luigi/issues/2880

TychonautVII avatar Apr 12 '21 14:04 TychonautVII

A better improvement actually might be to just leave the remote name as is

TychonautVII avatar Apr 12 '21 18:04 TychonautVII

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If closed, you may revisit when your time allows and reopen! Thank you for your contributions.

stale[bot] avatar Jan 09 '22 01:01 stale[bot]

so what's up? Can we add some fixes in next releases

elessarelfstone avatar Apr 17 '23 18:04 elessarelfstone