filesystem_spec
filesystem_spec copied to clipboard
Dealing with an FTP server that doesn't support offsets
Running code like:
with fsspec.open(an_ftp_url) as fin:
while True:
buf = fin.read(16*1024)
if not buf:
break
raises a error_temp: 426 Connection closed; transfer aborted.
here:
https://github.com/fsspec/filesystem_spec/blob/62bb12e681b9e3dbd25df3991fd71552ea7654ee/fsspec/implementations/ftp.py#L314
The server doesn't support the rest
argument.
From the docs: https://docs.python.org/3/library/ftplib.html#ftplib.FTP.transfercmd it sounds like this is not uncommon:
If the server does not recognize the REST command, an [error_reply](https://docs.python.org/3/library/ftplib.html#ftplib.error_reply) exception will be raised. If this happens, simply call [transfercmd()](https://docs.python.org/3/library/ftplib.html#ftplib.FTP.transfercmd) without a rest argument.
I'm curious if you have any suggestions on how do get an FTP server like this to work with fsspec?
The obvious answer of just reading the entire buffer into memory buf = fin.read()
is not satisfying. I'd like to be able to copy the file in chunks to a destination and not load the entire thing into memory.
Without REST, the only thing you can do it read the whole file in one go. So, cat
or get
would work fine. If you wanted open
to work, for the sake of your application logic, you could pass cache_type="all"
- I think that should work.