ws upstreams don't work: permanent disconnects/reconnects

[PDVJAM]

Hey! Each time I enable ws for upstreams infura or alchemy, I see only connection errors in the logs. But it works with my own geth node. dshackle version is 0.12.

2022-03-07 09:17:28.116 | INFO  |         WsConnection | Connecting to WebSocket: wss://eth-mainnet.g.alchemy.com/v2/xxx
2022-03-07 09:17:28.642 | INFO  |         WsConnection | Reconnect to wss://eth-mainnet.g.alchemy.com/v2/xxx in 100ms...
2022-03-07 09:17:28.643 | INFO  |         WsConnection | Disconnected from wss://eth-mainnet.g.alchemy.com/v2/xxx

dshackle config:

      - id: alchemy
        chain: ethereum
        role: fallback
        options:
          disable-validation: true
        connection:
          ethereum:
            rpc:
              url: "https://eth-mainnet.g.alchemy.com/v2/${ALCHEMY_KEY}"
            ws:
              url: "wss://eth-mainnet.g.alchemy.com/v2/${ALCHEMY_KEY}"

What can be wrong and how to fix that? Would be glad for the help.

[splix]

Can you check which the requests it makes and what responses it gets? With Wireshark maybe?

[PDVJAM]

Can you check which the requests it makes and what responses it gets? With Wireshark maybe?

That is interesting, but how to decrypt TLS in case of dshackle? I have tried to set SSLKEYLOGFILE via docker-compose, but the file is empty.

[splix]

I see, that's a problem. You cannot just decrypt the connection and see what's going on.

You said you also use it with Infura. Do you see the same problem with Infura? Asking because I use Infura for testing and it works fine with it, but have no experience with Alchemy and maybe it has some difference in protocol.

Also, can you check that you don't have any additional security settings. For Infura, it may be a JWT auth, specific Origins with CORS, or User Agent. Please make sure none of them is enabled in the Settings panel.

[PDVJAM]

I see, that's a problem. You cannot just decrypt the connection and see what's going on.

You said you also use it with Infura. Do you see the same problem with Infura? Asking because I use Infura for testing and it works fine with it, but have no experience with Alchemy and maybe it has some difference in protocol.

Also, can you check that you don't have any additional security settings. For Infura, it may be a JWT auth, specific Origins with CORS, or User Agent. Please make sure none of them is enabled in the Settings panel.

You were right, Infura had an origin validation. Now fixed and works. But Alchemy has "allowed all" and reconnects every 100ms.

[splix]

You can configure Origin for a WS connection, btw. See docs.

I think something similar is the reason for Alchemy, but I have no suggestion on what to try. But if you'll figure it out, please leave a comment here

[PDVJAM]

You can configure Origin for a WS connection, btw. See docs.

I think something similar is the reason for Alchemy, but I have no suggestion on what to try. But if you'll figure it out, please leave a comment here

No, with Alchemy it's something different. Everything is allowed in my settings, screenshot

[nlordell]

We ran into similar issues with Alchemy, and its related to it not supporting net_peers RPC request, which causes the WebSocket connection to "restart". See https://github.com/emeraldpay/dshackle/issues/155.

The issue seems to be that disable-validation: true is not respected for WebSocket upstreams.

However, it does respect validate-peers (and validate-syncing, but that isn't needed), which solved this for me on version [email protected] (didn't check for other versions).

cluster:
  upstreams:
    - id: alchemy
      chain: ethereum
      options:
        validate-peers: false
      connection:
        ethereum:
          ws:
            url: "wss://eth-mainnet.g.alchemy.com/v2/${ALCHEMY_API_KEY}"

[PDVJAM]

@nlordell Hey! I use the same 0.13.1 version. Just tried you solution, and now I don't see any disconnects/reconnects, but one warning instead:

2023-03-01 18:49:34.605 | WARN | WsConnection | Failed to process WS message. class java.lang.IllegalStateException: State is not ready

Do you see the same?

      - id: alchemy
        chain: ethereum
        role: secondary
        priority: 20
        options:
          validate-peers: false
        connection:
          ethereum:
            rpc:
              url: "https://eth-mainnet.g.alchemy.com/v2/${ALCHEMY_KEY}"
            ws:
              url: "wss://eth-mainnet.g.alchemy.com/v2/${ALCHEMY_KEY}"

[nlordell]

I do see those warnings as well every 30s or so. I mentioned it here: https://github.com/emeraldpay/dshackle/issues/155#issuecomment-1301928024 although I haven't looked more closely at the issue.