No SSH connection possible "Error connecting to web socket"

[timonbirk-witt]

Stratos Version

4.4.0

Frontend Deployment type

• [ x ] Cloud Foundry Application (cf push)
• [ ] Kubernetes, using a helm chart
• [ ] Docker, single container deploying all components
• [ ] npm run start
• [ ] Other (please specify below)

Backend (Jet Stream) Deployment type

• [ x ] Cloud Foundry Application (cf push)
• [ ] Kubernetes, using a helm chart
• [ ] Docker, single container deploying all components
• [ ] Other (please specify below)

Expected behaviour

SSH access to apps is possible. Stratos uses the correct id.

Actual behaviour

For several apps we can't create a ssh connection (via instances -> SSH) with stratos. We only get the error message "Error connecting to web socket".

After some debugging we discovered that Stratos is requesting with a "wrong" application id. Here are the logentries from the scheduler/ssh-proxy service in cloud foundry:

{​​​​​​​"timestamp":"2021-02-09T10:06:34.516532713Z","level":"error","source":"ssh-proxy","message":"ssh-proxy.authentication-failed","data":{​​​​​​​"error":"ssh: no auth passed yet","user":"cf:94c2a04d-eabc-44f9-a31f-0e9d666e9aec/[email protected]"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:06:34.517828483Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.cf-authenticate.authenticate-starting","data":{​​​​​​​"session":"264"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:06:34.858569794Z","level":"error","source":"ssh-proxy","message":"ssh-proxy.cf-authenticate.fetching-app-failed","data":{​​​​​​​"ResponseBody":{​​​​​​​}​​​​​​​,"StatusCode":"404 Not Found","app":"94c2a04d-eabc-44f9-a31f-0e9d666e9aec/0","error":"Fetching application data failed","principal":"99d98a36-8c29-416d-a424-f19289409848","session":"264","username":"me@mycompany"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:06:34.858785507Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.cf-authenticate.authenticate-finished","data":{​​​​​​​"session":"264"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:06:34.858832877Z","level":"error","source":"ssh-proxy","message":"ssh-proxy.authentication-failed","data":{​​​​​​​"error":"Fetching application data failed","user":"cf:94c2a04d-eabc-44f9-a31f-0e9d666e9aec/[email protected]"}​​​​​​​}​​​​​​​

So the app id "94c2a04d-eabc-44f9-a31f-0e9d666e9aec/0" could not be found.

Running the "cf cli" command for the same application results in

{"timestamp":"2021-02-09T10:26:08.465319920Z","level":"error","source":"ssh-proxy","message":"ssh-proxy.authentication-failed","data":{​​​​​​​"error":"ssh: no auth passed yet","user":"cf:4e874674-df2b-4eb1-ab62-ca90a3f4db91/0"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:26:08.467275889Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.cf-authenticate.authenticate-starting","data":{​​​​​​​"session":"738"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:26:08.751682714Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.cf-authenticate.app-access-success","data":{​​​​​​​"app":"4e874674-df2b-4eb1-ab62-ca90a3f4db91/0","principal":"99d98a36-8c29-416d-a424-f19289409848","session":"738","username":"me@mycompany"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:26:08.751719507Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.cf-authenticate.authenticate-finished","data":{​​​​​​​"session":"738"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:26:08.751752850Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.authentication-attempted","data":{​​​​​​​"user":"cf:4e874674-df2b-4eb1-ab62-ca90a3f4db91/0"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:26:08.767173858Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.handle-connection.new-client-conn.connected-to-backend","data":{​​​​​​​"backend-address":"10.176.213.8:61075","session":"737.1"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:26:08.773429884Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.handle-connection.from-client.proxy-global-requests.started","data":{​​​​​​​"session":"737.2.1"}​​​​​​​}​​​​​​​

and I can ssh into the app.

Seems like the cf cli (version 7.1.0) is using another id.

Having a closer look into the application we can see that cf cli is using the "process id" (4e874674-df2b-4eb1-ab62-ca90a3f4db91) and stratos the app id (94c2a04d-eabc-44f9-a31f-0e9d666e9aec).

VCAP_APPLICATION={"cf_api":"https://api.mycompany","limits":{"fds":16384,"mem":384,"disk":1024},"application_name":"myapp","application_uris":["myapp.mycompany"],"name":"myapp","space_name":"myspace","space_id":"d64180a4-e607-419f-92f9-d95c96ea7759","organization_id":"8611e8b0-ff63-4b76-9017-df69e8166331","organization_name":"myorg","uris":["myapp.mycompany"],"process_id":"4e874674-df2b-4eb1-ab62-ca90a3f4db91","process_type":"web","application_id":"94c2a04d-eabc-44f9-a31f-0e9d666e9aec","version":"43abc7e3-f40a-43b6-b8c4-9a88f0cd4ebd","application_version":"43abc7e3-f40a-43b6-b8c4-9a88f0cd4ebd"}

Maybe the difference is because the app was deployed with the CF 7 rolling app deployment (cf push APP-NAME --strategy rolling)?

Other applications (which we can access with stratos and cf cli) don't have different ids as app id and process id.

[richard-cox]

CF Apps deployed via CF CLI v7 will use v3 of the CF API (which brings in/exposes the process id). We've been waiting a long time for v3 to be completed and it's not currently supported by Stratos.