dshackle Turbo-Geth Websocket error

This one is kind of strange. I have a Turbo-Geth node configured and sync'd but if I use websocket I get the following error:

2020-23-08 20:37:56.518 | INFO  |  ConfiguredUpstreams | Using Ethereum upstream, at http://192.168.15.34:8545, ws://192.168.15.34:8545
2020-23-08 20:37:56.531 | INFO  | WsFactory$EthereumWs | Connecting to WebSocket: ws://192.168.15.34:8545
io.netty.handler.codec.http.websocketx.WebSocketHandshakeException: Invalid handshake response getStatus: 200 OK
	at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker13.verify(WebSocketClientHandshaker13.java:191)
	at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker.finishHandshake(WebSocketClientHandshaker.java:216)
	at io.netty.handler.codec.http.websocketx.WebSocketClientProtocolHandshakeHandler.channelRead(WebSocketClientProtocolHandshakeHandler.java:57)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
2020-23-08 20:37:56.606 | INFO  |     EthereumUpstream | Configured for Ethereum
2020-23-08 20:37:56.642 | INFO  | entMultistreamHolder | Upstream turbogeth with chain ETHEREUM has been added

I'm not sure if this is something strange with dshackle or if Turbo-Geth is doing something weird.

The Turbo-Geth team hasn't done extensive WebSocket testing, I know that much. But the error is strange to me.

Maybe you have an idea?

Aug 23 '20 20:08 MysticRyuujin

I'm thinking that maybe Turbo-Geth is missing some RPC endpoint that dshackle requires, because it comes back as unavailable if I only use HTTP too. :disappointed:

Aug 23 '20 20:08 MysticRyuujin

Hi, thank you for the finding. There was a severe bug in the WS code, which I had fixed recently (not release yet). But this one may be unrelated. I'm trying to make a release later today with other bugfixes, so let's try with the next version, and we'll see.

Aug 23 '20 23:08 splix

I just published version 0.8.2, please try it. Not sure about this issue though, but if fixed couple of other issues you reported.

Aug 24 '20 01:08 splix

No change with latest version

Aug 24 '20 02:08 MysticRyuujin

Ok, thanks.

Just looked at the stacktrace, it seems to be more complex that I thought. It happens inside WS codec implementation (WebSocketClientHandshaker13), not even Dshackle. I guess I'll need a Turbo Geth to debug it.

Aug 24 '20 02:08 splix

If you DM me on Gitter, I can open a port up to my TurboGeth instance for you. I just need your source IP to stick on the firewall.

Aug 24 '20 02:08 MysticRyuujin

I'm just not sure when I can focus on this bug. I guess it's not high priority, because it can continue with HTTP RPC, right? I can take a look at it later then

Aug 24 '20 20:08 splix

Correct

Aug 24 '20 21:08 MysticRyuujin

I have an update!

Turbo-Geth now supports WebSocket...apparently they did not before, which explains why it didn't work haha.

However, it appears that dshackle gets upset at Turbo-Geth not supporting eth_subscribe, kills the WebSocket, and returns to JSON RPC.

dshackle: {"jsonrpc":"2.0","method":"eth_subscribe","params":["newHeads"],"id":1}

Turbo-Geth: {"jsonrpc":"2.0","id":1,"error":{"code":-32601,"message":"no \"newHeads\" subscription in eth namespace"}}\n

In the screenshot you can see that WebSocket stops after that reply, and then a new TCP connection is made and HTTP is used.

The HTTP 400 Bad Request is because turning on WebSocket in Turbo-Geth currently disables JSON RPC :laughing: I was told they'll fix it.

Sep 01 '20 22:09 MysticRyuujin

Kind of good news :) The main purpose to use WS for Dshackle is to subscribe to the new blocks (newHeads), hope they'll add it

Sep 01 '20 23:09 splix

I know there are other efficiency gains from using WebSocket, is it not worth using WebSocket even without the subscription?

Sep 01 '20 23:09 MysticRyuujin

You're right, it probably makes sense to use WebSocket for all requests. It relies on HTTP only because of the historical reasons, was easier to start/debug/etc. Hope to change it in one of the future releases, not sure when

Sep 01 '20 23:09 splix