heartbeat with force for inter-operability

[hyangilam]

Problem/use case that this feature is meant to solve/enable Dear Joffrey, These days, I found that server side send stomp error frame even though stomp client(krossbow) does send messages(not heartbeat). I observed that krossbow does not send heartbeat if there is any tx message at heartbeat period time. According to stomp 1.2 spec, server side should not have sent this error because any message can replace heartbeat

Describe the solution you'd like For inter-operability with these server(which does not support the spec fully), I want for krossbow to have additional heartbeat configuration. Forced transmission

Describe alternatives you've considered

(Is there a way -maybe inconvenient- to achieve your goal at the moment, or a workaround?)

[joffrey-bion]

Hi, thanks for opening this issue!

This is an interesting use case. It's super weird that this server requires heartbeats in addition to messages, as heartbeats are mostly here to inform that the connection is still alive, which obviously is unnecessary if messages are being sent. Could you please provide the name of this server implementation? Or a link to the server's project? It would be useful to add this information to the documentation if I add a way to customize the heartbeat strategy, so people understand why it's there.

Also, maybe it's worth trying to fix the server regarding this.

[joffrey-bion]

Also, please clarify what the server expects here. STOMP heartbeats are not really special frames, the only part about the actual data in the specification is the following:

Regarding the heart-beats themselves, any new data received over the network connection is an indication that the remote end is alive. In a given direction, if heart-beats are expected every <n> milliseconds:

• the sender MUST send new data over the network connection at least every milliseconds

• if the sender has no real STOMP frame to send, it MUST send an end-of-line (EOL)

• if, inside a time window of at least milliseconds, the receiver did not receive any new data, it MAY consider the connection as dead

• because of timing inaccuracies, the receiver SHOULD be tolerant and take into account an error margin

Does that mean the server you're interacting with requires a single EOL (0x0A byte) to be sent on the wire every N milliseconds regardless of the presence of other frames?

[khnointotheworld]

Hello @joffrey-bion

I have tried to figure out what is actual behavior about heart-beat in krossbow. I also added cases flutter stomp to compare.

I have used below general backend server environment.

Spring boot framework kotlin version "2.6.3" org.springframework:spring-messaging:5.3.15 org.springframework.messaging.simp.stomp

1. Flutter stomp client

• It always sending heart-beat to server as configuration even if there is transaction of stomp frame for user defined message. I have checked with pcap in backend server. Yeah. It doesn't need to send to heartbeat if there is others as heartbeat 1.1 or 1.2 standard. So, Spring server can be detected client is gone.

2. Krossbow

• It is exactly working as heart-beat standard. In my pcap dump, it skips several heart-beat during transaction is on going. But spring boot server seems to check always heart-beat EOL, it ignores user message in the connection. So, few skips makes spring boot misunderstand client is gone.

@joffrey-bion Do you have some plan to provide option to keep sending heart-beat without skip for poor framework.:) I also tried HeartBeatTolerance in krossbow, but it doesn't work, spring always obsess heart-beat for client's health.

[joffrey-bion]

Thank you very much for the additional information. This is quite valuable.

it skips several heart-beat during transaction is on going

Do you mean an actual STOMP transaction? Or just regular messages?

But spring boot server seems to check always heart-beat EOL, it ignores user message in the connection

Could you please share the STOMP configuration you are using on Spring Boot side, and on Krossbow side? (a sample project would be awesome!). I'd like to reproduce the problem, because I have myself used Krossbow to talk to a STOMP Spring boot server with heartbeats and never had this problem.

From my experience, Spring has worked with heart beats according to spec: it didn't fail if no EOL was received, as long as messages kept coming. The only thing I had noticed is that it was quite slow to send its own heartbeats sometimes and had a few seconds of delay compared to the expected time, this is the reason why I added heartbeat tolerance in Krossbow.

Also, could you please share the error you get on server side (if any)? And what do you get exactly on client side?

Do you have some plan to provide option to keep sending heart-beat without skip for poor framework.:)

Well, Spring is a very very popular framework, so if it does behave like you say, I will definitely implement some workaround. However, I am not entirely sure that this is actually what's happening, and this is why I would like to reproduce the problem before considering this feature.

[joffrey-bion]

Hey @khnointotheworld, it would be of great help if you could provide the additional info that I mentioned in my previous message! 🙏

[khnointotheworld]

Hi @joffrey-bion, Sorry for delayed response. Finally my team member found some description and fixed this issue. I hope others also refer this when "stomp connection broker closed" has been occurred.

public StompBrokerRelayRegistration setTaskScheduler(@Nullable TaskScheduler taskScheduler) { this.taskScheduler = taskScheduler; return this; }

Some STOMP clients (e.g. stomp-js) always send heartbeats at a fixed rate but others (Spring STOMP client) do so only when no other messages are sent. However messages with a non-broker destination prefix aren't forwarded and as a result the broker may deem the connection inactive. When this TaskScheduler is set, it is used to reset a count of the number of messages sent from client to broker since the beginning of the current heartbeat period. This is then used to decide whether to send a heartbeat to the broker when ignoring a message with a non-broker destination prefix.

We added task scheduler for handling of heart beat missing cases.

override fun configureMessageBroker(registry: MessageBrokerRegistry) { val heartbeatTaskScheduler = ThreadPoolTaskScheduler() heartbeatTaskScheduler.poolSize = 40 // as of now hardcoded heartbeatTaskScheduler.setThreadNamePrefix("wss-heartbeat-thread-") heartbeatTaskScheduler.initialize() registry.setApplicationDestinationPrefixes(applicationDestinationPrefix) .enableStompBrokerRelay(destinationPrefix) .setSystemHeartbeatReceiveInterval(15000).setSystemHeartbeatSendInterval(10000) // as of now hardcoded with bigger value 15000 .setTaskScheduler(heartbeatTaskScheduler) .setRelayHost(host) .setRelayPort(port.toInt()) .setClientLogin(userName) .setClientPasscode(password) .setSystemLogin(userName) .setSystemPasscode(password) }

[joffrey-bion]

Thanks for the detailed analysis! Glad to hear that this fixed your issue. I'll close this for now until a use case that really requires forcing heart beats comes up.