postfacto
postfacto copied to clipboard
ActionCable payload string too long
We've got postfacto running and it is working well, until the content of the retro becomes large.
At this point the realtime actions all stop working. Upon looking at the logs it would appear that we've hit the payload limit for postgres notify, 8000 bytes, as documented here: https://www.postgresql.org/docs/9.0/sql-notify.html
D, [2021-03-22T11:21:18.838280 #12] DEBUG -- : [4d9032339701837d1a7219c716a26aa9] [ActionCable] Broadcasting to retros:Z2lkOi8vcmV0cm8tYXBwL1JldHJvLzQ: <very large json object>
I, [2021-03-22T11:21:18.862815 #12] INFO -- : [4d9032339701837d1a7219c716a26aa9] Completed 500 Internal Server Error in 51ms (ActiveRecord: 6.5ms | Allocations: 15201)
F, [2021-03-22T11:21:18.863712 #12] FATAL -- : [4d9032339701837d1a7219c716a26aa9]
[4d9032339701837d1a7219c716a26aa9] PG::InvalidParameterValue (ERROR: payload string too long
):
[4d9032339701837d1a7219c716a26aa9]
[4d9032339701837d1a7219c716a26aa9] app/channels/retros_channel.rb:39:in `broadcast'
[4d9032339701837d1a7219c716a26aa9] app/controllers/items_controller.rb:81:in `broadcast'
[4d9032339701837d1a7219c716a26aa9] app/controllers/items_controller.rb:53:in `vote'
Is there a suggested workaround for this? Or is there something wrong in our configuration of postfacto?
Presumably you're using Postgres for ActionCable rather than Redis (i.e. you've set the USE_POSTGRES_FOR_ACTION_CABLE
environment variable)? Perhaps this is an undocumented limitation of that adapter (cc @mshahat related to #334).
thanks @textbook , I've not enabled USE_POSTGRES_FOR_ACTION_CABLE
on my heroku setup. i am happy to try it and feedback later.
@davet1985 is it the content of a retro item? or action item ? so i can test.
Thanks,
Thanks @textbook and @mshahat for your quick responses!
Yes we're using Postgres for ActionCable. The issue can be triggered by adding lots of items to a retro, with long text.
Just to update this issue, I've added a redis instance and disable USE_POSTGRES_FOR_ACTION_CABLE
which solves the problem. It seems there is a very low size limit in the current implementation when using postgres for ActionCable.
That was my suspicion, thanks for the update @davet1985. For now I think we'll have to document this as a limitation of the Postgres adapter, I think properly fixing this would be a major change to how Postfacto uses the websocket data.