asyncssh.sftp.SFTPFailure: The message [<filename>] is not extractable!

[jobevers]

When trying to read a file from an SFTP server, I'm getting an error saying asyncssh.sftp.SFTPFailure: The message [<filename>] is not extractable!

  File "[...]//lib/python3.7/site-packages/fsspec/asyn.py", line 86, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "[...]/lib/python3.7/site-packages/fsspec/asyn.py", line 66, in sync
    raise return_result
  File "[...]//lib/python3.7/site-packages/fsspec/asyn.py", line 26, in _runner
    result[0] = await coro
  File "[...]//lib/python3.7/site-packages/sshfs/utils.py", line 49, in _method
    return await wrapped_meth(*args, **kwargs)
  File "[...]//lib/python3.7/site-packages/asyncssh/sftp.py", line 3076, in read
    size = (await self._end()) - offset
  File "[...]//lib/python3.7/site-packages/asyncssh/sftp.py", line 3024, in _end
    attrs = await self.stat()
  File "[...]//lib/python3.7/site-packages/asyncssh/sftp.py", line 3222, in stat
    return await self._handler.fstat(self._handle, flags)
  File "[...]/lib/python3.7/site-packages/asyncssh/sftp.py", line 2733, in fstat
    FXP_FSTAT, String(handle), flag_bytes))
  File "[...]//lib/python3.7/site-packages/asyncssh/sftp.py", line 2460, in _make_request
    result = self._packet_handlers[resptype](self, resp)
  File "[...]/lib/python3.7/site-packages/asyncssh/sftp.py", line 2476, in _process_status
    raise exc

I'm able to manually run the sftp cli, access the server and GET the file.

This is with version 2.11.0, python 3.7 on ubuntu 18.04.

[jobevers]

This looks related: https://github.com/paramiko/paramiko/pull/562

[ronf]

Thanks for the report -- I've never run across servers which behave this way.

I agree that the traceback above suggests it may be the same problem as what you linked in Paramiko, and it's possibly that the high-level copy function I have in AsyncSSH could run into a similar problem that I might be able to fix by passing along the srcattrs I query at the beginning to avoid later calling stat() on the SFTP handle. However, I'm not sure that such a change will help you, as it looks like you're not using the high-level AsyncSSH functions to do the read.

From the traceback, it appears that sshfs is what is (indirectly) calling stat() on the open SFTP handle in your case. More specifically, it's calling read() with an offset but no size argument and so AsyncSSH has to call _end() which does the stat() to calculate the total size to read. To avoid that stat() call, I think the sshfs package would need to be changed to get the size information and pass an appropriate size into the read() call. The AsyncSSH SFTPClientFile object which read() is being called on doesn't have access to the path, so it can't do a path-based stat() at that level. I think the only option is to calculate the size in the caller.

[ronf]

Looking more closely at the high-level API code in AysncSSH for get/put/copy, I don't think any changes are required to avoid the issue mentioned here when those calls are used. The existing stat() calls in that code are only on pathnames, and not on an open SFTP handle. From reading over the discussion in the paramiko link above, it looks like stat() calls on a path should be safe.

As noted above, this doesn't prevent the problem completely, if calling code does it own stat() on a handle, or triggers that indirectly with something like a read() on a handle without specifying a size. It looks like doing a seek() on a handle could also trigger this when doing a seek relative to the end of the file, but I'm guessing such an operation would probably be unlikely on one of these "one time access" files.

To fix this in the calling code, I think you'd need to do a stat() on the path to get the file's size before opening the file, and then pass in this size when the call is made to read(). If you knew a maximum size you wanted to read, you could potentially pass that in without doing a stat(), and I believe AsyncSSH should return a short read with whatever remains in the file if there are fewer bytes than what was requested.