ArrayIndexOutOfBoundsException while downloading the file

[anujkhandelwal1990]

trafficstars

Hi,

We recently faced "java.lang.ArrayIndexOutOfBoundsException" Exception while downloading the file from SFTP server. This happens intermittently. Few observations:

• Mainly it happens when the Thread which downloads the file is running from a long time (~10 minutes)
• The thread has to download ~100 files one by one.
• Once we see this error, it keeps on happening for few minutes but it happens for all the files downloaded with the thread.
• It becomes normal after sometime and we are able to download files after that.

Did someone also faced the similar error? What could cause this ? What are the workaround for this ?

[anurag-08-09]

Just to add here we also faced similar issue in last of couple of days.

[anujkhandelwal1990]

Adding complete stack trace for reference.

We are using version 0.23.0

manager.SFTPFilePatternMatcherManager.matchPatterns():258 ERROR]: Error while downloading file from server java.lang.ArrayIndexOutOfBoundsException: -5 at net.schmizz.sshj.common.Buffer.putUInt32(Buffer.java:320) at net.schmizz.sshj.transport.Encoder.encode(Encoder.java:83) at net.schmizz.sshj.transport.TransportImpl.write(TransportImpl.java:449) at net.schmizz.sshj.connection.channel.ChannelOutputStream$DataBuffer.flush(ChannelOutputStream.java:102) at net.schmizz.sshj.connection.channel.ChannelOutputStream$DataBuffer.flush(ChannelOutputStream.java:70) at net.schmizz.sshj.connection.channel.ChannelOutputStream.flush(ChannelOutputStream.java:182) at net.schmizz.sshj.sftp.SFTPEngine.transmit(SFTPEngine.java:297) at net.schmizz.sshj.sftp.SFTPEngine.request(SFTPEngine.java:130) at net.schmizz.sshj.sftp.SFTPEngine.doRequest(SFTPEngine.java:136) at net.schmizz.sshj.sftp.SFTPEngine.stat(SFTPEngine.java:263) at net.schmizz.sshj.sftp.SFTPEngine.stat(SFTPEngine.java:217) at net.schmizz.sshj.sftp.SFTPFileTransfer.download(SFTPFileTransfer.java:70) at net.schmizz.sshj.sftp.SFTPFileTransfer.download(SFTPFileTransfer.java:59) at net.schmizz.sshj.sftp.SFTPClient.get(SFTPClient.java:229) at net.schmizz.sshj.sftp.StatefulSFTPClient.get(StatefulSFTPClient.java:182) at company.munshi.transfers.service.impl.SFTPService.getFile(SFTPService.java:505) at com.company.transporter.manager.SFTPFilePatternMatcherManager.matchPatterns(SFTPFilePatternMatcherManager.java:253) at com.company.transporter.processor.SFTPDataProcessor.matchPatterns(SFTPDataProcessor.java:74) at com.company.transporter.processor.DataProcessor.run(DataProcessor.java:131) at com.company.transporter.web.TransporterSFTPController.downloadFiles(TransporterSFTPController.java:91) at sun.reflect.GeneratedMethodAccessor170.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852) at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at com.company.commons.servlet.utils.filter.RequestMDCFilter.doFilter(RequestMDCFilter.java:65) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at company.propeq.webcommons.filter.EffectiveUserFilter.doFilter(EffectiveUserFilter.java:284) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at company.propeq.webcommons.filter.KerberosHttpFilter.authenticate(KerberosHttpFilter.java:471) at company.propeq.webcommons.filter.KerberosHttpFilter.doFilter(KerberosHttpFilter.java:370) at company.propeq.webcommons.filter.AuthenticationFilter.runFilter(AuthenticationFilter.java:821) at company.propeq.webcommons.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:663) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at com.company.commons.security.filter.CORSFilter.doFilter(CORSFilter.java:134) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)

[hierynomus]

It looks like the write position of the buffer overflowed. Can you give more detailed logging of the chain of events?

[vanseverk]

Hi there,

We seem to be getting the same issue when using the SFTPClient.put method. This does not happen every time the method is called, and no pattern could be deduced.

private void sendSftpFile(String fileType, String date, String sequenceNumber, String content) throws IOException {
    InMemorySourceFile inMemoryFile = createInMemoryFile(fileType, date, sequenceNumber, content);
    sftpClient.put(inMemoryFile, SFTP_FOLDER + "/" + inMemoryFile.getName());
}

java.lang.ArrayIndexOutOfBoundsException: -5 at net.schmizz.sshj.common.Buffer.putUInt32(Buffer.java:320) at net.schmizz.sshj.transport.Encoder.encode(Encoder.java:102) at net.schmizz.sshj.transport.TransportImpl.write(TransportImpl.java:449) at net.schmizz.sshj.connection.channel.ChannelOutputStream$DataBuffer.flush(ChannelOutputStream.java:102) at net.schmizz.sshj.connection.channel.ChannelOutputStream$DataBuffer.flush(ChannelOutputStream.java:70) at net.schmizz.sshj.connection.channel.ChannelOutputStream.flush(ChannelOutputStream.java:182) at net.schmizz.sshj.sftp.SFTPEngine.transmit(SFTPEngine.java:297) at net.schmizz.sshj.sftp.SFTPEngine.request(SFTPEngine.java:130) at net.schmizz.sshj.sftp.SFTPEngine.doRequest(SFTPEngine.java:136) at net.schmizz.sshj.sftp.SFTPEngine.stat(SFTPEngine.java:263) at net.schmizz.sshj.sftp.SFTPEngine.stat(SFTPEngine.java:217) at net.schmizz.sshj.sftp.SFTPFileTransfer$Uploader.isDirectory(SFTPFileTransfer.java:285) at net.schmizz.sshj.sftp.SFTPFileTransfer$Uploader.upload(SFTPFileTransfer.java:181) at net.schmizz.sshj.sftp.SFTPFileTransfer$Uploader.access$100(SFTPFileTransfer.java:166) at net.schmizz.sshj.sftp.SFTPFileTransfer.upload(SFTPFileTransfer.java:64) at net.schmizz.sshj.sftp.SFTPClient.put(SFTPClient.java:244) at com.company.backend.domain.ftp.gateway.SftpGatewayImpl.sendSftpFile(SftpGatewayImpl.java:70)

[matenhagen]

java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)Caused by: java.lang.ArrayIndexOutOfBoundsException: Index -5 out of bounds for length 512 at net.schmizz.sshj.common.Buffer.putUInt32(Buffer.java:320) at net.schmizz.sshj.transport.Encoder.encode(Encoder.java:102) at net.schmizz.sshj.transport.TransportImpl.write(TransportImpl.java:449) at net.schmizz.sshj.connection.channel.ChannelOutputStream$DataBuffer.flush(ChannelOutputStream.java:102) at net.schmizz.sshj.connection.channel.ChannelOutputStream$DataBuffer.flush(ChannelOutputStream.java:70) at net.schmizz.sshj.connection.channel.ChannelOutputStream.flush(ChannelOutputStream.java:182) at net.schmizz.sshj.sftp.SFTPEngine.transmit(SFTPEngine.java:297) at net.schmizz.sshj.sftp.SFTPEngine.request(SFTPEngine.java:130) at net.schmizz.sshj.sftp.SFTPEngine.doRequest(SFTPEngine.java:136) at net.schmizz.sshj.sftp.SFTPEngine.remove(SFTPEngine.java:203) at net.schmizz.sshj.sftp.SFTPClient.rm(SFTPClient.java:125) at

Same issue when removing a file.

[vladimirlagunov]

I've seen this error quite a lot when something tries to use an already disconnected session. It must be a marker of some other problem that caused the disconnection. I managed to get rid of this error in our code completely. Try to check SSHClient.isConnected before calling SFTP operations, collect and handle all thrown IOExceptions carefully.

By the way, when I looked at the code that throws the error, I didn't find an easy way to fix that.