api-layer icon indicating copy to clipboard operation
api-layer copied to clipboard

TN3270 doesn't work on 7554 port

Open Josema-Kyndryl opened this issue 2 years ago • 30 comments

We migrated to Zowe v2.2 under z/OS v2.4

By accessing https://zosa.tor.es.ibm.com:7554/zlux/ui/v1/ all the applications (API Catalog, the different explorers, ..) work fine, but the TN3270 (TLS) fails.

imagen

However, if we access TN3270 from https://zosa.tor.es.ibm.com:7556 it works perfectly:

imagen

Any idea?

Josema-Kyndryl avatar Sep 30 '22 10:09 Josema-Kyndryl

@1000TurquoisePogs, could you, please, take a look at this issue? It might be related to zLux.

anton-brezina avatar Sep 30 '22 13:09 anton-brezina

Note - this issue seems to be actively discussed in the OM slack.

anton-brezina avatar Sep 30 '22 13:09 anton-brezina

Sorry, Tono

I've practically included the same information about our issue on both channels (Github & Slack)

Josema-Kyndryl avatar Sep 30 '22 14:09 Josema-Kyndryl

I'm sure the slack conversation is deleted now so hope there wasnt something important in there. I don't know the cause of this but happy to help. Here's what I know so far:

  • the code involved doesn't change often see server and client ... i'm not aware of any meaningful change in a long time because it's been stable and without feature requests.

  • Joe W had a 1xxx issue that seemed to be solved by clearing the cookies out of his browser. Is this issue really just a server responding back with HTTP "not authorized" over a websocket and causing the browser's websocket logic to throw an error because it's not websocket traffic?

  • 1xxx codes come from the browser, rather than zowe code, so we don't know why the browser decided to do it. but we can enable tracing on the terminal proxy server via

components:
  app-server:
    logLevels:
      org.zowe.terminal.proxy.*: 3

1000TurquoisePogs avatar Feb 23 '23 15:02 1000TurquoisePogs

Hi everyone

I have added in zowe.yaml the sentences that you indicate and restarted ZWESLSTC.

I accessed Zowe Desktop by the 7554 port and, as always, when trying to access the TN3270 I got the error 1003

But I have not seen any debug message related to this neither in the Syslog of the task nor in the files of the directory .../logs

However, after accessing by 7556 port messages of this type appear.... imagen

Josema-Kyndryl avatar Mar 02 '23 12:03 Josema-Kyndryl

@1000TurquoisePogs , after moving from IBM to Kyndryl we have stopped using Slack (the corporation has decided to use Microsoft Teams)

Josema-Kyndryl avatar Mar 02 '23 12:03 Josema-Kyndryl

That essentially means that network activity is reaching the server through 7556 and not through 7554. A possible cause is authentication error, and clearing your browser cookies has worked for some people with the same issue.

As for slack, we mean the Zowe slack. It is not related to IBM or Kyndryl and can be used without a corporate slack, as it is a website. It is where you can find us, but messages are not kept after a few weeks. Various links on it are here https://www.zowe.org/contribute.html

1000TurquoisePogs avatar Mar 03 '23 11:03 1000TurquoisePogs

Thanks a lot. Indeed it must be something of my FireFox because I have tried it from Safari and it works fine

Josema-Kyndryl avatar Mar 06 '23 13:03 Josema-Kyndryl

Hi, re-oening this issue as it fails on firefox and all browsers, when you sign on, and then go away for a day and come back. Even if you have signed off (which clears the Zowe SSO cookie) there are other cookies left, and even though you can log back into the Zowe Desktop, the TN3270 and VT Terminal will do a timeout wait. It looks as though there is something in the cookies left there that is stopping web sockets. Clearing cookies manually does help, but I know customers running into this. I'm happy to re-create this and help debug the problem, but I don't know exactly what is making Zowe's web socket I/O go back which occurs consistently every time.

Joe-Winchester avatar Mar 27 '23 08:03 Joe-Winchester

@Joe-Winchester Are you able to replicate on at least Zowe 2.7?

balhar-jakub avatar Apr 24 '23 06:04 balhar-jakub

Hi Jakub, yes I can re-create this consistently. There is a call with the client on Wednesday at 1pm EST where we're hoping to do some more debugging. If anyone from the community wants to join that'd be great, as the only workaround we have is to clear cookies which works for a few seconds then the desktop behind APIML (7754) stops working again.

Joe-Winchester avatar Apr 24 '23 17:04 Joe-Winchester

Hi All. To try to debug the connection issue, we used Postman to run WebSocket requests to ports 7554 and 7556. The error messages for each port are different:

  • Port 7554: "Invalid WebSocket frame: invalid status code 1005"
  • Port 7556: "1005 no status received: Missing status code even though one was expected"

screenshots below: Screenshot 2023-04-28 at 8 57 23 AM Screenshot 2023-04-28 at 8 57 35 AM

Jack-Billings-IBM avatar Apr 28 '23 16:04 Jack-Billings-IBM

Hi, Some issues with the websockets implementation were fixed in #2914 and will be released with Zowe 2.9. We did some testing with @Joe-Winchester and so far we are unable to replicate a similar condition with this version. Convenience paxes can be downloaded here with the unreleased changes if it's possible for you to verify https://zowe.jfrog.io/artifactory/list/libs-release-local/org/zowe/nightly/v2/

pablocarle avatar May 22 '23 07:05 pablocarle

Hi,

I've reviewed server logs and found no issues in the gateway routing of websocket frames.

@1000TurquoisePogs I see some custom websocket error codes in the server logs such as this one:

2023-06-05 14:54:42.465 <ZWEAGW1:HttpClient@17dbbfe1-28:33555445> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyClientHandler) afterConnectionClosed(session=JettyWebSocketSession[id=a7a91af7-2854-e9ca-05a4-d1387ae56ed8, uri=wss://<host>:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],status=CloseStatus[code=4000, reason=Closed by user])

Which is the Gateway detecting a session being closed. I found that code 4000 comes from the tn3270 and vt terminals code but I'm not sure how to get more diagnostics from these services (https://github.com/zowe/tn3270-ng2/blob/205833bf1dda851f8cd1da89d6dea17562b0af80/webClient/src/app/terminal.ts#L118)

I think this issue should be transferred to zlux

pablocarle avatar Jun 09 '23 11:06 pablocarle

To turn on maximum logging if needed, here's how you do it:

server-side, we have a websocket-to-TCP proxy. In the zowe.yaml, you can configure server-side logging of that plugin like so:

  app-server:
    enabled: true
    port: 7556
    dataserviceAuthentication:
      defaultAuthentication: "saf"
    logLevels:
      org.zowe.terminal.proxy.*: 2

The default level is 2. 3, 4 & 5 are varying levels of debug. Use 5 with caution! It prints might print actual terminal bytes, which would tend to contain passwords you have typed.

In the browser: The terminal receives the bytes from that proxy, and also sends keystrokes to the proxy. The terminal in the browser has waaaay too much logging, so don't turn it all on at once unless you want to slow the page to a crawl.

https://github.com/zowe/tn3270-ng2/wiki/Loggers

The way you turn on the logging is as seen in the screenshot in the wiki: you need to use the browser's developer tools console (you can press f12, but not in the terminal because that'd send PF12!) also, make sure the terminal is open BEFORE you set these in the browser, or it wont do anything.

1000TurquoisePogs avatar Jun 09 '23 13:06 1000TurquoisePogs

@1000TurquoisePogs It seems the issue reported is also related to the VT terminal use. In https://github.com/zowe/vt-ng2 I don't see a wiki like with the TN3270. What are the loggers that need to be enabled in app-server to get details of what this terminal is doing?

pablocarle avatar Jun 27 '23 08:06 pablocarle

Adding here logs with both Gateway websocket debugging enabled and terminal logs:

2023-06-26 13:19:56.240 <ZWEAGW1:https-jsse-nio-0.0.0.0-7554-exec-2:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyServerHandler) handleMessage(session=StandardWebSocketSession[id=88765552-f94f-bdc8-3039-21984e1459fd, uri=wss://mvst.host.com:7554/zlux/ws/v1/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"VT_I..], byteCount=30, last=true])
2023-06-26 13:19:56.240 <ZWEAGW1:https-jsse-nio-0.0.0.0-7554-exec-2:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketRoutedSession) sendMessageToServer(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"VT_I..], byteCount=30, last=true])
2023-06-26 13:19:56.245 <ZWEAGW1:HttpClient@cfe6b335-29:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyClientHandler) handleMessage(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"TELN..], byteCount=32, last=true])
2023-06-26 13:19:56.267 <ZWEAGW1:HttpClient@cfe6b335-30:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyClientHandler) handleMessage(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"TELN..], byteCount=48, last=true])
2023-06-26 13:19:56.269 <ZWEAGW1:HttpClient@cfe6b335-29:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyClientHandler) handleMessage(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"TELN..], byteCount=32, last=true])
2023-06-26 13:19:56.272 <ZWEAGW1:HttpClient@cfe6b335-30:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyClientHandler) handleMessage(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"TELN..], byteCount=556, last=true])
2023-06-26 13:19:56.294 <ZWEAGW1:https-jsse-nio-0.0.0.0-7554-exec-9:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyServerHandler) handleMessage(session=StandardWebSocketSession[id=88765552-f94f-bdc8-3039-21984e1459fd, uri=wss://mvst.host.com:7554/zlux/ws/v1/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"VT_I..], byteCount=46, last=true])
2023-06-26 13:19:56.294 <ZWEAGW1:https-jsse-nio-0.0.0.0-7554-exec-9:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketRoutedSession) sendMessageToServer(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"VT_I..], byteCount=46, last=true])
2023-06-26 13:19:57.675 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:411) ZWED0277I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Received host data. Length=18
2023-06-26 13:19:57.676 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:400) ZWED0276I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Websocket sending client message. Length=52
2023-06-26 13:19:57.407 <ZWEAGW1:RibbonApacheHttpClientConfiguration.connectionManagerTimer:67109295> ZWESVUSR DEBUG (o.a.h.i.c.PoolingHttpClientConnectionManager) Closing expired connections
2023-06-26 13:19:57.677 <ZWEAGW1:HttpClient@cfe6b335-29:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyClientHandler) handleMessage(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"TELN..], byteCount=52, last=true])


2023-06-26 13:19:56.242 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:265) ZWED0269I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Websocket client message received. Length=30
2023-06-26 13:19:56.242 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:394) ZWED0275I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Writing to host socket. Length=1
2023-06-26 13:19:56.243 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:411) ZWED0277I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Received host data. Length=2
2023-06-26 13:19:56.244 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:400) ZWED0276I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Websocket sending client message. Length=32
2023-06-26 13:19:56.266 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:411) ZWED0277I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Received host data. Length=15
2023-06-26 13:19:56.266 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:400) ZWED0276I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Websocket sending client message. Length=48
2023-06-26 13:19:56.268 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:411) ZWED0277I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Received host data. Length=2
2023-06-26 13:19:56.268 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:400) ZWED0276I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Websocket sending client message. Length=32
2023-06-26 13:19:56.269 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:411) ZWED0277I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Received host data. Length=395
2023-06-26 13:19:56.269 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:400) ZWED0276I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Websocket sending client message. Length=556
2023-06-26 13:19:56.295 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:265) ZWED0269I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Websocket client message received. Length=46
2023-06-26 13:19:56.295 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:394) ZWED0275I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Writing to host socket. Length=15
2023-06-26 13:19:56.240 <ZWEAGW1:https-jsse-nio-0.0.0.0-7554-exec-2:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyServerHandler) handleMessage(session=StandardWebSocketSession[id=88765552-f94f-bdc8-3039-21984e1459fd, uri=wss://mvst.host.com:7554/zlux/ws/v1/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"VT_I..], byteCount=30, last=true])
2023-06-26 13:19:56.240 <ZWEAGW1:https-jsse-nio-0.0.0.0-7554-exec-2:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketRoutedSession) sendMessageToServer(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"VT_I..], byteCount=30, last=true])
2023-06-26 13:19:56.245 <ZWEAGW1:HttpClient@cfe6b335-29:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyClientHandler) handleMessage(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"TELN..], byteCount=32, last=true])
2023-06-26 13:19:56.267 <ZWEAGW1:HttpClient@cfe6b335-30:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyClientHandler) handleMessage(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"TELN..], byteCount=48, last=true])
2023-06-26 13:19:56.269 <ZWEAGW1:HttpClient@cfe6b335-29:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyClientHandler) handleMessage(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"TELN..], byteCount=32, last=true])
2023-06-26 13:19:56.272 <ZWEAGW1:HttpClient@cfe6b335-30:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyClientHandler) handleMessage(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"TELN..], byteCount=556, last=true])
2023-06-26 13:19:56.294 <ZWEAGW1:https-jsse-nio-0.0.0.0-7554-exec-9:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyServerHandler) handleMessage(session=StandardWebSocketSession[id=88765552-f94f-bdc8-3039-21984e1459fd, uri=wss://mvst.host.com:7554/zlux/ws/v1/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"VT_I..], byteCount=46, last=true])
2023-06-26 13:19:56.294 <ZWEAGW1:https-jsse-nio-0.0.0.0-7554-exec-9:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketRoutedSession) sendMessageToServer(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"VT_I..], byteCount=46, last=true])
2023-06-26 13:19:57.675 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:411) ZWED0277I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Received host data. Length=18
2023-06-26 13:19:57.676 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:400) ZWED0276I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Websocket sending client message. Length=52
2023-06-26 13:19:57.677 <ZWEAGW1:HttpClient@cfe6b335-29:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyClientHandler) handleMessage(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],message=TextMessage payload=[{"t":"TELN..], byteCount=52, last=true])
2023-06-26 13:20:01.392 <ZWED:67109955> ZWESVUSR INFO (org.zowe.terminal.proxy:vtdata,terminalProxy.js:210) ZWED0097I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Websocket closed. Total remaining terminals connected: 0
2023-06-26 13:20:01.392 <ZWED:67109955> ZWESVUSR INFO (org.zowe.terminal.proxy:vtdata,terminalProxy.js:221) ZWED0100I - Total VT terminals connected: 0
2023-06-26 13:20:01.393 <ZWED:67109955> ZWESVUSR DEBUG (org.zowe.terminal.proxy:vtdata,terminalProxy.js:607) ZWED0282I - [Host=omvst.host.com, Port=23, ClientIP=10.120.10.214] Error: Host closed socket
2023-06-26 13:20:01.387 <ZWEAGW1:HttpClient@cfe6b335-30:67109295> ZWESVUSR DEBUG (o.z.a.g.w.WebSocketProxyClientHandler) afterConnectionClosed(session=JettyWebSocketSession[id=07202884-77f4-c982-132f-68152e54c5d2, uri=wss://mvst.host.com:7556/ZLUX/plugins/org.zowe.terminal.vt/services/terminalstream/_current/],status=CloseStatus[code=4000, reason=Closed by user])

There's no indication of issues in the API Gateway. I think it could use analysis from the Zlux team. Also note the different domain, for Gateway it's mvst.host.com and for the terminal this is omvst.host.com. At this moment I'm unsure if this could lead to problems.

pablocarle avatar Jul 11 '23 07:07 pablocarle

@1000TurquoisePogs It seems the issue reported is also related to the VT terminal use. In https://github.com/zowe/vt-ng2 I don't see a wiki like with the TN3270. What are the loggers that need to be enabled in app-server to get details of what this terminal is doing?

Same loggers, I'll just copy-paste the wiki.

1000TurquoisePogs avatar Jul 18 '23 16:07 1000TurquoisePogs

Can someone comment is this was addressed in Zowe 2.15? I don't see a mention of it in the Release Notes

armstro avatar Mar 12 '24 15:03 armstro

Hi @armstro , the 2.15 release allows the ability to increase a number of timeout values related to websockets. The suspicion is (from looking at customer environments where TN3270 timed out when going via APIML and worked on a direct connection) that this may help.
The release notes for 2.15 do have a mention of this, however it does look as though the docs need updating to include an example which I believe is

zowe:
  components:
    gateway:
      apiml:
        server:
          websocket:
            connectTimeout: 15000
            stopTimeout: 30000
            asyncWriteTimeout: 60000
            maxIdleTimeout:3600000

By playing with these values and increasing the timeout value the hope is that the websocket timeout can be increased to avoid the issue that customers were experiencing, but we need more field testing to be sure. @pablocarle was the one who did the 2.5 system demo showcase of the feature so he'll have more details.

Joe-Winchester avatar Mar 12 '24 22:03 Joe-Winchester

Hi @armstro , the 2.15 release allows the ability to increase a number of timeout values related to websockets. The suspicion is (from looking at customer environments where TN3270 timed out when going via APIML and worked on a direct connection) that this may help. The release notes for 2.15 do have a mention of this, however it does look as though the docs need updating to include an example which I believe is

zowe:
  components:
    gateway:
      apiml:
        server:
          websocket:
            connectTimeout: 15000
            stopTimeout: 30000
            asyncWriteTimeout: 60000
            maxIdleTimeout:3600000

By playing with these values and increasing the timeout value the hope is that the websocket timeout can be increased to avoid the issue that customers were experiencing, but we need more field testing to be sure. @pablocarle was the one who did the 2.5 system demo showcase of the feature so he'll have more details.

This is in fact in the API ML changelog: https://github.com/zowe/api-layer/blob/v2.x.x/CHANGELOG.md#2150--zowe-2150-2024-02-21 Not sure why it would be missing from the 2.15 Zowe but we can review that.

Just a small correction in the yaml, the example would be like this:

gateway:
        server:
            webSocket:
                maxIdleTimeout: 3600000
                connectTimeout: 15000
                stopTimeout: 30000
                asyncWriteTimeout: 60000

Note the capital S in webSocket and the server level just below gateway and not under apiml. Be aware that the server level exists by default in the zowe.yaml, so in most cases only the webSocket section would be the one you'd add.

pablocarle avatar Mar 13 '24 09:03 pablocarle

Hi everyone.

We currently have zowe v2.14 installed, so we can't make those changes to the configuration requested by you.

I've requested an upgrade to v2.15; As soon as I'm there, I'll be able to check if the terminals work fine with these parameters

Josema-Kyndryl avatar Mar 13 '24 10:03 Josema-Kyndryl

Hello,

We installed v2.15 and have put the following in the yaml: webSocket: maxIdleTimeout: 3600000 connectTimeout: 15000 stopTimeout: 30000 asyncWriteTimeout: 60000

We then tested VT Terminal and TN3270 and still see some users experiencing connection issues.

gidon-lando avatar Mar 18 '24 15:03 gidon-lando

Hello,

We installed v2.15 and have put the following in the yaml: webSocket: maxIdleTimeout: 3600000 connectTimeout: 15000 stopTimeout: 30000 asyncWriteTimeout: 60000

We then tested VT Terminal and TN3270 and still see some users experiencing connection issues.

Hi @gidon-lando,

Could you provide the yaml's gateway section for validation?

The values you've provided are the defaults. As such, it will be no different than with Zowe v2.14, please try increasing these values. In particular the connect timeout.

pablocarle avatar Mar 18 '24 15:03 pablocarle

Hey @pablocarle,

Sure, here is the gateway section:

  gateway:
    enabled: true
    port: 7554
    debug: false

    apiml:
      security:
        auth:
          provider: zosmf
          zosmf:
            jwtAutoconfiguration: auto
            serviceId: zosmf
        authorization:
          endpoint:
            enabled: false
          provider: ""
        x509:
          enabled: false
    server:
      internal:
      # gateway supports internal connector
        enabled: false
        port: 7550
        ssl:
          enabled: false
          # internal connector can use different certificate
          # certificate:
          #   keystore:
          #     alias: ""
      webSocket:
      # added 031324 to increase websocket timeout interval to resolve
      # VT terminal and TN3270 connection problem
        maxIdleTimeout: 3600000
        connectTimeout: 15000
        stopTimeout: 30000
        asyncWriteTimeout: 60000

I didn't realize these were the default values. I'll increase connectTimeout and try again.

gidon-lando avatar Mar 18 '24 15:03 gidon-lando

Hi Everyone

After migrate to Zowe v2.15 and fix the parameters suggested, the 3270 terminal work fine in our system

Josema-Kyndryl avatar Mar 27 '24 11:03 Josema-Kyndryl

I have a same issue as others. components:

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

gateway: enabled: true port: 7554 debug: true #debug: false server: websocket: maxIdleTimeout: 3600000 connectTimeout: 15000 stopTimeout: 30000 asyncWriteTimeout: 60000

cfl0000 avatar Jun 21 '24 21:06 cfl0000

I have a same issue as others. components:

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

gateway: enabled: true port: 7554 debug: true #debug: false server: websocket: maxIdleTimeout: 3600000 connectTimeout: 15000 stopTimeout: 30000 asyncWriteTimeout: 60000

Hi @cfl0000, Have you tried longer values for the connectTimeout? Keep in mind 15000 (15 seconds) it's the default so far and it may be short for the time it can take to start TN3270 or VT Terminal in Zowe Desktop.

pablocarle avatar Jun 24 '24 08:06 pablocarle

Good day; I changed it to 30000, and it does not work either.

On Mon, Jun 24, 2024 at 4:51 AM Pablo Carle @.***> wrote:

I have a same issue as others. components:

gateway: enabled: true port: 7554 debug: true #debug: false server: websocket: maxIdleTimeout: 3600000 connectTimeout: 15000 stopTimeout: 30000 asyncWriteTimeout: 60000

Hi @cfl0000 https://github.com/cfl0000, Have you tried longer values for the connectTimeout? Keep in mind 15000 (15 seconds) it's the default so far and it may be short for the time it can take to start TN3270 or VT Terminal in Zowe Desktop.

— Reply to this email directly, view it on GitHub https://github.com/zowe/api-layer/issues/2603#issuecomment-2185957324, or unsubscribe https://github.com/notifications/unsubscribe-auth/BJK7QBQ2PNFO47CAJSSF4PLZI7MZJAVCNFSM6AAAAAAQZUKCXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBVHE2TOMZSGQ . You are receiving this because you were mentioned.Message ID: @.***>

cfl0000 avatar Jun 24 '24 15:06 cfl0000

We found out that there is one more thing at the very least that may be impacting this behavior and that's buffer size for the communication. E.g. if there are too many large cookies, the communication may fail due to the amount of data transmitted. The default is only 4k.

@cfl0000 With the next minor release, we will provide a way to configure this behaviour. Specifically, it's managed by the requestBufferSize property. The current default value was 4k; the snippet below proposes changing it to 8k. If you want to try sooner, you can use the nightly build, which should already have this change.

zowe:
  components:
    gateway:
          connectTimeout: 15000
          stopTimeout: 30000
          asyncWriteTimeout: 60000
          maxIdleTimeout: 3600000
          maxIdleTimeout: 3600000
          requestBufferSize: 8192

balhar-jakub avatar Jun 25 '24 07:06 balhar-jakub