STOMP: Transaction storage length handling

[ikavgo]

Describe the bug

How it looks to me now - stomp transaction data stored in pdicts which are part of the process heap. Large binaries are not part of the process heap. The separate limit for them was added recently by the way - https://github.com/erlang/otp/issues/5889.

So with default +hmax set to 0 it appears like stomp connection can grow and grow. Also without https://github.com/erlang/otp/issues/5889. (which is +hmbs if I'm not mistaken) even with hmax set we can grow and grow.

Here is how transaction grows - https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbitmq_stomp/src/rabbit_stomp_processor.erl#L1036

Reproduction steps

import stomp
import random
import string

c = stomp.Connection([('127.0.0.1', 61613)])
c.connect('guest', 'guest', wait=True)
txid = c.begin()
body = ''.join(random.choices(string.ascii_letters, k=2*1024*1024))
for n in range(100000000):
    c.send('/topic/test', body, transaction=txid)

Expected behavior

Number of transaction actions is limited and/or connection process memory, including binaries is limited.

Additional context

I don't think stomp respects watermarks and blocks publishing.

[michaelklishin]

Since 3.13 will require Erlang 26, we should be able to use those flags if they make sense.

[ikavgo]

https://github.com/rabbitmq/rabbitmq-server/issues/8870 for amqp