While downloading multiple files, FTPException(Failure) is thrown

[cct-dai-sato]

trafficstars

I am trying to read multiple RemoteFiles into InputStreams as described here.

The code works fine up to some point but suddenly fails by throwing FTPException as below.

...
2020-05-14 22:36:38,640 [debug] n.s.s.s.SFTPEngine - Sending Request{1049;OPEN}
2020-05-14 22:36:38,640 [debug] n.s.s.c.c.Window$Remote - Consuming by 51 down to 2044833
2020-05-14 22:36:38,652 [debug] n.s.s.c.c.Window$Local - Consuming by 17 down to 1619133
2020-05-14 22:36:38,652 [debug] n.s.s.s.PacketReader - Received HANDLE packet
2020-05-14 22:36:38,652 [debug] n.s.s.s.SFTPEngine - Sending Request{1050;OPEN}
2020-05-14 22:36:38,652 [debug] n.s.s.c.c.Window$Remote - Consuming by 51 down to 2044782
2020-05-14 22:36:38,663 [debug] n.s.s.c.c.Window$Local - Consuming by 17 down to 1619116
2020-05-14 22:36:38,663 [debug] n.s.s.s.PacketReader - Received HANDLE packet
2020-05-14 22:36:38,663 [debug] n.s.s.s.SFTPEngine - Sending Request{1051;OPEN}
2020-05-14 22:36:38,663 [debug] n.s.s.c.c.Window$Remote - Consuming by 51 down to 2044731
2020-05-14 22:36:38,674 [debug] n.s.s.c.c.Window$Local - Consuming by 17 down to 1619099
2020-05-14 22:36:38,674 [debug] n.s.s.s.PacketReader - Received HANDLE packet
2020-05-14 22:36:38,674 [debug] n.s.s.s.SFTPEngine - Sending Request{1052;OPEN}
2020-05-14 22:36:38,674 [debug] n.s.s.c.c.Window$Remote - Consuming by 51 down to 2044680
2020-05-14 22:36:38,685 [debug] n.s.s.c.c.Window$Local - Consuming by 17 down to 1619082
2020-05-14 22:36:38,685 [debug] n.s.s.s.PacketReader - Received HANDLE packet
2020-05-14 22:36:38,685 [debug] n.s.s.s.SFTPEngine - Sending Request{1053;OPEN}
2020-05-14 22:36:38,685 [debug] n.s.s.c.c.Window$Remote - Consuming by 51 down to 2044629
2020-05-14 22:36:38,696 [debug] n.s.s.c.c.Window$Local - Consuming by 28 down to 1619054
2020-05-14 22:36:38,696 [debug] n.s.s.s.PacketReader - Received STATUS packet
2020-05-14 22:36:38,697 [error] m.p.CustomJsonHttpErrorHandler - 7fn7ij306, Execution exception, [SFTPException: Failure], {}
net.schmizz.sshj.sftp.SFTPException: Failure
	at net.schmizz.sshj.sftp.Response.error(Response.java:140)
	at net.schmizz.sshj.sftp.Response.ensurePacketTypeIs(Response.java:117)
	at net.schmizz.sshj.sftp.SFTPEngine.open(SFTPEngine.java:143)
	at net.schmizz.sshj.sftp.SFTPEngine.open(SFTPEngine.java:149)
	at net.schmizz.sshj.sftp.SFTPEngine.open(SFTPEngine.java:154)

In what case, does the client receive the Status packet and end up in failure?

[hierynomus]

Take a look at the server side logging. There might be more information there.

[cct-dai-sato]

I do not have access to the log... (I have no control over the SFTP server settings or etc)

If this was the server-side problem, what would it be the cause?

[Kordishal]

I ran into the same issue. After some investigation, I figured out that it was because the files were not closed. The connection failed to open new files each time exactly after 1021 files were opened and not closed.

The reason why they were not closed is, that the RemoteFileInputStream does not actually implement a close function. It just uses the empty InputStream close. I think the RemoteFileInputStream should close the underlying RemoteResource.

The same goes for the RemoteFileOutputStream which simply flushes all the writes, but does not close the RemoteResource.

[hierynomus]

Actually because the RemoteFile is opened, you should be closing that. The problem is if you would close the RemoteFile if you closed the RemoteFileOutputStream is that you wouldn't be able to call setAttributes after you closed the RemoteFileOutputStream. This would thus give unwanted side-effects.

[Kordishal]

I suppose that is a legitimate use case for OUTPUT Stream. But I am mostly concerned about the InputStream.

It is standard behaviour for a resource to be closed when you no longer need it. If I close an Input Stream I except all of the resources to be close.

Your RemoteFileInputStream should then not implement Closable. Since it is not actually closeable.

Or at least make a note of this in the documentation that this is non-standard behaviour.

Because I wanted to abstract all of your implementations behind a small library, but now I have to import the dependency directly in every project anyway for compilation.

[hierynomus]

@Kordishal I do get your point. However, I do have a different point of view.

You open the RemoteFile, you close it. It is a Closeable resource, so no magic happens, you are responsible. This is also how most Java libraries operate, you don't close resources that are not under your scope of control. There is no telling what the user of the library is going to do with the resource next.

We could add a SFTPClient.getOutputStream(path) or SFTPClient.getInputStream(path) that would open and return the Stream in a single call, in that case the library is responsible for the closing of the RemoteResource.

If you use library functions, such as SFTPClient.upload/download, managing the RemoteResource and Streams is taken care of.

In absence of the methods above, if you want to abstract away the call in your own library, I would suggest you return a custom InputStream that closes the underlying file upon the close() call, and which delegates all the other calls to the RemoteFileInputStream

[Kordishal]

I see. I suppose that makes sense. Then I will do as you say.

Thanks for the quick responses.