Dealing with an FTP server that doesn't support offsets

[jobevers]

Running code like:

with fsspec.open(an_ftp_url) as fin:
    while True:
        buf = fin.read(16*1024)
        if not buf:
            break

raises a error_temp: 426 Connection closed; transfer aborted. here: https://github.com/fsspec/filesystem_spec/blob/62bb12e681b9e3dbd25df3991fd71552ea7654ee/fsspec/implementations/ftp.py#L314

The server doesn't support the rest argument. From the docs: https://docs.python.org/3/library/ftplib.html#ftplib.FTP.transfercmd it sounds like this is not uncommon:

If the server does not recognize the REST command, an [error_reply](https://docs.python.org/3/library/ftplib.html#ftplib.error_reply) exception will be raised. If this happens, simply call [transfercmd()](https://docs.python.org/3/library/ftplib.html#ftplib.FTP.transfercmd) without a rest argument.

I'm curious if you have any suggestions on how do get an FTP server like this to work with fsspec?

The obvious answer of just reading the entire buffer into memory buf = fin.read() is not satisfying. I'd like to be able to copy the file in chunks to a destination and not load the entire thing into memory.

[martindurant]

Without REST, the only thing you can do it read the whole file in one go. So, cat or get would work fine. If you wanted open to work, for the sake of your application logic, you could pass cache_type="all" - I think that should work.