File resource download hangs when trying to download file

[machristie]

Description

I'm running download_http.py to generate a download url for a file resource. When I click on the generated URL to download the file, it just hangs.

Steps to Reproduce

I created a SCP file resource

...
  {
    "type": "SCP",
    "resourceId":  "remote-ssh-resource",
    "resourceMode": "FILE",
    "resourcePath": "/tmp/test1/solution_summary.json",
    "storageId" : "remote-ssh-storage"
  },

Then I generate the download URL with a python script (modified download_http.py):

import grpc
import MFTApi_pb2
import MFTApi_pb2_grpc

channel = grpc.insecure_channel('localhost:7004')
stub = MFTApi_pb2_grpc.MFTApiServiceStub(channel)
download_request = MFTApi_pb2.HttpDownloadApiRequest(
    # sourceStoreId ="remote-ssh-storage",
    sourceResourceId = "remote-ssh-resource",
    # sourceResourceId =  "remote-ssh-dir-resource",
    # sourceStoreId ="remote-ssh-dir-resource",
    # sourcePath= "/tmp/test1/foo",
    sourceToken = "local-ssh-cred",
    sourceType= "SCP",
    targetAgent = "agent0",
    # mftAuthorizationToken = ""
    )

result = stub.submitHttpDownload(download_request)
print(result)

When I run the script I get:

$ python download_http.py 
url: "http://localhost:3333/4978023d-c807-46fc-989b-8f550d18021d"
targetAgent: "agent0"

When I load the URL in a browser, the browser just spins and never finishes loading the file.

Expected Behaviour

Loading the URL in the browser would result in a file download.

Your Environment

• mft branch or release version used: develop
• Operating system and version: macOS Big Sur

Additional Context

By using logging statements, I know that it gets to trying to trying to create a session in SCPReceiver.java

https://github.com/apache/airavata-mft/blob/develop/transport/scp-transport/src/main/java/org/apache/airavata/mft/transport/scp/SCPReceiver.java#L103

Apparently it is stuck on this line in SCPTransportUtil:

https://github.com/apache/airavata-mft/blob/develop/transport/scp-transport/src/main/java/org/apache/airavata/mft/transport/scp/SCPTransportUtil.java#L37

because I can log the line before but none of my logging after this line prints.

Also, I know that the SCP configuration is good since I'm able to fetch metadata for the resource:

request = MFTApi_pb2.FetchResourceMetadataRequest(

    resourceId= "remote-ssh-resource",
                                        resourceType = "SCP",
                                        resourceToken = "local-ssh-cred",
                                        targetAgentId = "agent0",
                                        )
response = stub.getFileResourceMetadata(request)
print(response)

[machristie]

I've done some debugging and I've found that for some reason the private key is invalid. But also, when the exception is thrown it is causing the thread pool mechanism to just hang, but I'm not sure why. I'll continue investigating.

[machristie]

It turns out it was a private key format issue. Downloading a file uses the SCPReceiver/SCPSender classes and these classes use the JSch library instead of the ssh4j library. JSch doesn't support the newer format of OpenSSH private keys, see https://stackoverflow.com/questions/53134212/invalid-privatekey-when-using-jsch.

I was able to convert my private key to the older format by using this:

ssh-keygen -p -f local-ssh-cred -m pem

The -m pem argument tells openssh to write private key in the classic PEM format.

This private key is now useable with ssh4j and JSch.

[machristie]

@DImuthuUpe I have this fixed for now but I think there are still a few issues here:

• exception handling needs to be fixed in HttpServerHandler. The pullStatusFuture results in an Exception when the key file is not a supported format and then the connection just hangs.
• SCPReceiver/SCPSender use JSch but the rest of the codebase uses ssh4j. Would it be possible to switch SCPReceiver/SCPSender to ssh4j too?