Sending error can't modify frozen String

[Wouter0100]

Describe the bug

Once in a while an e-mail is not send and instead we are faced with a FrozenError: can't modify frozen String error. A manual or automated retry normally fixefs it.

To Reproduce

I have unfortunately not found a way to reproduce it yet. It seems to happen once in a while.

Expected behaviour

Proper sending of the e-mail.

Screenshots

Environment details

Ubuntu 20.04.3, Postal v2.1.2 in Docker

Additional information/context

[9] [2022-09-13T10:27:13.191] INFO -- : [CE3RLMAKUD] Started processing \u001b[34mUnqueueMessageJob\u001b[0m job
[9] [2022-09-13T10:27:13.338] INFO -- : [CE3RLMAKUD] Lock acquired for queued message 1179032
[9] [2022-09-13T10:27:13.969] INFO -- : [CE3RLMAKUD] Found 0 associated messages to process at the same time (batch key: outgoing-hotmail.co.uk)
[9] [2022-09-13T10:27:13.995] INFO -- : [CE3RLMAKUD] [98::11381 1179032] Got queued message with exclusive lock
[9] [2022-09-13T10:27:14.292] INFO -- : [CE3RLMAKUD] [98::11381 1179032] Parsing message content as it hasn't been parsed before
[9] [2022-09-13T10:29:16.534] INFO -- : [CE3RLMAKUD] [98::11381 1179032] Internal error: FrozenError: can't modify frozen String
[9] [2022-09-13T10:29:16.534] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /opt/postal/app/lib/postal/message_db/delivery.rb:53:in `force_encoding'
[9] [2022-09-13T10:29:16.534] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /opt/postal/app/lib/postal/message_db/delivery.rb:53:in `webhook_hash'
[9] [2022-09-13T10:29:16.535] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /opt/postal/app/lib/postal/message_db/delivery.rb:44:in `send_webhooks'
[9] [2022-09-13T10:29:16.535] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /opt/postal/app/lib/postal/message_db/delivery.rb:11:in `create'
[9] [2022-09-13T10:29:16.535] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /opt/postal/app/lib/postal/message_db/message.rb:128:in `create_delivery'
[9] [2022-09-13T10:29:16.535] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /opt/postal/app/app/jobs/unqueue_message_job.rb:419:in `block in perform'
[9] [2022-09-13T10:29:16.535] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /opt/postal/app/app/jobs/unqueue_message_job.rb:30:in `each'
[9] [2022-09-13T10:29:16.535] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /opt/postal/app/app/jobs/unqueue_message_job.rb:30:in `perform'
[9] [2022-09-13T10:29:16.535] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /opt/postal/app/lib/postal/worker.rb:55:in `receive_job'
[9] [2022-09-13T10:29:16.535] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /opt/postal/app/lib/postal/worker.rb:88:in `block in join_queue'
[9] [2022-09-13T10:29:16.536] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /usr/local/bundle/gems/bunny-2.14.4/lib/bunny/consumer.rb:56:in `call'
[9] [2022-09-13T10:29:16.536] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /usr/local/bundle/gems/bunny-2.14.4/lib/bunny/channel.rb:1768:in `block in handle_frameset'
[9] [2022-09-13T10:29:16.536] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /usr/local/bundle/gems/bunny-2.14.4/lib/bunny/consumer_work_pool.rb:108:in `block (2 levels) in run_loop'
[9] [2022-09-13T10:29:16.537] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /usr/local/bundle/gems/bunny-2.14.4/lib/bunny/consumer_work_pool.rb:103:in `loop'
[9] [2022-09-13T10:29:16.537] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /usr/local/bundle/gems/bunny-2.14.4/lib/bunny/consumer_work_pool.rb:103:in `block in run_loop'
[9] [2022-09-13T10:29:16.538] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /usr/local/bundle/gems/bunny-2.14.4/lib/bunny/consumer_work_pool.rb:102:in `catch'
[9] [2022-09-13T10:29:16.538] INFO -- : [CE3RLMAKUD] [98::11381 1179032] /usr/local/bundle/gems/bunny-2.14.4/lib/bunny/consumer_work_pool.rb:102:in `run_loop'
[9] [2022-09-13T10:29:16.549] INFO -- : [CE3RLMAKUD] [98::11381 1179032] Queued message was unlocked
[9] [2022-09-13T10:29:16.668] INFO -- : [CE3RLMAKUD] Finished processing \u001b[34mUnqueueMessageJob\u001b[0m job in 123.477191951s

Related to #1363, but as this one is closed I'm opening a new one.

[willpower232]

Thanks for the extra information, unfortunately I don't know enough ruby to understand this problem. Those retries are close together so I wonder how related they are.

[olegbliaher]

@Wouter0100 Copy-pasting my answer from #1363:

I had the same issue exactly when using Postal with rspamd, whenever an email gets marked as spam.

Modifying the line in question in lib/postal/message_db/delivery.rb to this seems to solve it:

:output => self.output.to_s+"".force_encoding('UTF-8').scrub,

[Wouter0100]

That's so lovely, thanks for figuring this out @olegbliaher! Would it me possible you'd open a Merge Request for this change?

Thanks! :open_hands:

[olegbliaher]

@Wouter0100 sure, but I'm not sure it would ever be approved, feels like the issue is somewhere higher up, it's unclear for me why output is nil in certain situations.

[catphish]

This should be fixed by f0a8aca6e10064fb16daefff9e22dcc20a831868 It's valid for output to be nil, and to_s was supposed to handle this case, but since Ruby 2.7, nil.to_s is frozen, so we need to duplicate the string before modifying it to make it safe for the database, which is a good idea anyway.

[0x7466]

Is there a new version release planned for this?

[an0nz]

Thanks for the fix, I ran into this issue after updating to the current release (v2.1.4), for now I have manually patched the fix into the clusters containers, looking forward to the next release to avoid building a custom image.

[rbustos]

Thanks for the fix, I ran into this issue after updating to the current release (v2.1.4), for now I have manually patched the fix into the clusters containers, looking forward to the next release to avoid building a custom image.

Same issue after update. Do you have a step by step procedure to fix this in prod?

[tbryant-vygr]

@rbustos if you are running a container environment then you can do the following to manually patch the bug docker ps .... Get the postal container id's docker exec -it <container-id> bash ~/app$ nano lib/postal/message_db/delivery.rb Find line 51: output: output.to_s.force_encoding("UTF-8").scrub.truncate(512), Change it to: output: output.to_s.dup.force_encoding("UTF-8").scrub.truncate(512), Save changes then exit the container

Refer to https://github.com/postalserver/postal/commit/f0a8aca6e10064fb16daefff9e22dcc20a831868 for the fix I did this for all postal containers to be safe but it likely only needs to be done for workers

[rbustos]

I did this in same way.. however this version looks like bring with other issues. I've applied rollback to 2.1.2 sadly.

[zajinx]

hope this gets fixed

[almivision]

@rbustos if you are running a container environment then you can do the following to manually patch the bug docker ps .... Get the postal container id's docker exec -it <container-id> bash ~/app$ nano lib/postal/message_db/delivery.rb Find line 51: output: output.to_s.force_encoding("UTF-8").scrub.truncate(512), Change it to: output: output.to_s.dup.force_encoding("UTF-8").scrub.truncate(512), Save changes then exit the container

Refer to f0a8aca for the fix I did this for all postal containers to be safe but it likely only needs to be done for workers

Hello @tbryant-vygr, can you help me with this problem? I tried your steps.. but still not working..

FrozenError: can't modify frozen String: "" Support Ref: J-GYE8KWL7PH

[nicksellen]

I encountered this issue, and would be great if there is a new release which contains the fix.

In the meantime I patched the image, and pushed a new image, see https://github.com/nicksellen/postal-fix

[Trapulo]

I've the same problem every time I send email to a specific domain (and maybe others): still no release for an EXISTING patch of a problem reported more than one year ago? :(

[Trapulo]

@rbustos if you are running a container environment then you can do the following to manually patch the bug docker ps .... Get the postal container id's docker exec -it <container-id> bash ~/app$ nano lib/postal/message_db/delivery.rb Find line 51: output: output.to_s.force_encoding("UTF-8").scrub.truncate(512), Change it to: output: output.to_s.dup.force_encoding("UTF-8").scrub.truncate(512), Save changes then exit the container Refer to f0a8aca for the fix I did this for all postal containers to be safe but it likely only needs to be done for workers

Hello @tbryant-vygr, can you help me with this problem? I tried your steps.. but still not working..

FrozenError: can't modify frozen String: "" Support Ref: J-GYE8KWL7PH

I also tried but still I have the problem

[gigbuilder]

I followed the exact instructions above -- nothing -- restarted postal -- still broken

[BorjaRafolsMartinez]

Same here, happens when I set the credentials to "Hold messages from this credential"

[Jank1310]

Same here, happens when I set the credentials to "Hold messages from this credential" <= Same here :/

[an0nz]

If you are having trouble manually patching I suggest you change to using the image @nicksellen put up which is available at https://github.com/nicksellen/postal-fix

As for receiving the same error when set to hold messages, if you have applied the patch (f0a8aca correctly and still get that error then you will need to submit a bug report on how to reproduce it.

[nicksellen]

It would be really useful to have this fix released! @catphish you nicely fixed it back in May, but the last release is from April, so doesn't include the fix. Maybe you, @adamcooke, or @willpower232 can create a new release? 🙏

[pdifeo]

@rbustos if you are running a container environment then you can do the following to manually patch the bug docker ps .... Get the postal container id's docker exec -it <container-id> bash ~/app$ nano lib/postal/message_db/delivery.rb Find line 51: output: output.to_s.force_encoding("UTF-8").scrub.truncate(512), Change it to: output: output.to_s.dup.force_encoding("UTF-8").scrub.truncate(512), Save changes then exit the container

Refer to f0a8aca for the fix I did this for all postal containers to be safe but it likely only needs to be done for workers

For me too, after making the change the problem persists. I think you have to restart the worker. I don't know how to do that. If I restart the container the change is lost, clearly.

[damienlaguerre]

I've written a bash script to automate the fix on all containers. It patches the containers and restarts them.

#!/bin/bash

for CONTAINER_ID in $(docker ps -f "name=postal_" --format "{{.ID}}")
do
        echo -e "Patch container $CONTAINER_ID"
        docker exec -it "$CONTAINER_ID" sed -i -i '[email protected][email protected]_s.dup.force_encoding@g'  lib/postal/message_db/delivery.rb
        echo -e "Restart Container $CONTAINER_ID"
        docker restart "$CONTAINER_ID"
done

[pdifeodfware]

I've written a bash script to automate the fix on all containers. It patches the containers and restarts them.

#!/bin/bash

for CONTAINER_ID in $(docker ps -f "name=postal_" --format "{{.ID}}")
do
        echo -e "Patch container $CONTAINER_ID"
        docker exec -it "$CONTAINER_ID" sed -i -i '[email protected][email protected]_s.dup.force_encoding@g'  lib/postal/message_db/delivery.rb
        echo -e "Restart Container $CONTAINER_ID"
        docker restart "$CONTAINER_ID"
done

@damienlaguerre great .

thanks a lot. In my enviroment the name of containers are with "postal-" and not "postal_" .

[chrisdeeming]

I've written a bash script to automate the fix on all containers. It patches the containers and restarts them.

#!/bin/bash

for CONTAINER_ID in $(docker ps -f "name=postal_" --format "{{.ID}}")
do
        echo -e "Patch container $CONTAINER_ID"
        docker exec -it "$CONTAINER_ID" sed -i -i '[email protected][email protected]_s.dup.force_encoding@g'  lib/postal/message_db/delivery.rb
        echo -e "Restart Container $CONTAINER_ID"
        docker restart "$CONTAINER_ID"
done

All of my postal containers names were in the format of postal- rather than postal_ so I needed to adjust this to get it to work.

For anyone else that needs to run this script, it is designed to output information so if you run it and get no output, that might be why.

[nicksellen]

There's been a flurry of releases recently, so hopefully should be fixed in the latest released version! I haven't upgraded yet though.

(Or anything 2.1.6 and higher according to https://github.com/postalserver/postal/issues/2707#issuecomment-1921363116)