UTF-8 characters unserialize() error at offset

[faculezcano]

trafficstars

Hi! First of all, thanks for such a useful package.

Im having troubles with data unserialization from SQS. Looking arround a bit i found this forum post so i think it could be related to UTF-8 characters encoding to 1 character (possibly a ?) and php expecting 2 characters. At the end of the forum post says that setting the charset in the request headers utf characters will encode correctly. If that isnt possible the other approach could be base64 encode the data sent to sqs and decode it back.

Part of a stack trace:

{
   "class":"ErrorException",
   "message":"unserialize(): Error at offset 1000 of 2395 bytes",
   "code":0,
   "file":"/var/app/current/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php:53",
   "trace":[
      "/var/app/current/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php:53",
      "/var/app/current/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php:98",
      "/var/app/current/vendor/dusterio/laravel-aws-worker/src/Jobs/AwsJob.php:46",
      "/var/app/current/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:356",
      "/var/app/current/vendor/dusterio/laravel-aws-worker/src/Wrappers/Laravel6Worker.php:37",
      "/var/app/current/vendor/dusterio/laravel-aws-worker/src/Controllers/WorkerController.php:135",
      "/var/app/current/vendor/dusterio/laravel-aws-worker/src/Controllers/LaravelController.php:45",
      "/var/app/current/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:45",
      "/var/app/current/vendor/laravel/framework/src/Illuminate/Routing/Route.php:239",
      "/var/app/current/vendor/laravel/framework/src/Illuminate/Routing/Route.php:196",
      "/var/app/current/vendor/laravel/framework/src/Illuminate/Routing/Router.php:685",
      "/var/app/current/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:128",
      "/var/app/current/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:103",
      "/var/app/current/vendor/laravel/framework/src/Illuminate/Routing/Router.php:687",
      "/var/app/current/vendor/laravel/framework/src/Illuminate/Routing/Router.php:662",
      "/var/app/current/vendor/laravel/framework/src/Illuminate/Routing/Router.php:628",
      "/var/app/current/vendor/laravel/framework/src/Illuminate/Routing/Router.php:617"
   ]
}

[brad-tilmor]

I've been having the same issue for months, and it's been difficult to debug and test +1

[cbcoolbox]

We are running into unserialization issues also, for us it seems to be based around two spaces in a row. We haven't been able to figure out how or why this is happening. There are no special characters or anything, if we remove one of the spaces in the sqs message it works fine so two spaces is somehow an issue.

[MPJHorner]

Just upgraded to Laravel 8 and getting this issue. Any idea to find out which ones are an issue?

[MPJHorner]

We are running into unserialization issues also, for us it seems to be based around two spaces in a row. We haven't been able to figure out how or why this is happening. There are no special characters or anything, if we remove one of the spaces in the sqs message it works fine so two spaces is somehow an issue.

What do you mean "remove one of the sapces in the SQS message" ?

[brad-tilmor]

@dusterio Do you have any idea what the cause of this might be (or have you run into it yourself)?

Update on my end, I ended up implementing a base64_encode workaround, and that seemed to solve it for a while, but after more requests were processed, I started seeing the error come up again, albeit less frequently.

[cbcoolbox]

We are using beanstalk and we were able to talk to an AWS tech and they said they were able to reproduce this issue on their side. It apparently is some sort of bug with the daemon they have that reads from the sqs queue and shoves it into where the laravel worker can read it. We weren't giving a timeline on a fix or if they will even fix it but that's what we found out.

[Nilanth]

@cbcoolbox Any updates?

[JulioPablo]

I can also confirm the behavior specified by @cbcoolbox with the double spaces, I was able to mitigate it in some cases by sanitizing any string that goes in the job with preg_replace('/\s+/', ' ', $yourString); and that lowered the amount of instances where this error was thrown, but this is not ideal/possible in all cases.

[cbcoolbox]

@Nilanth No updates at all. We are now moving to docker and ECS to use SQS more directly to go around their daemon that's breaking everything. We have some silly hacky fixes similar to what @JulioPablo did for the time being. technically the base64 encoding of your message would be the best "fix" we had that work also but feels terrible to code this way. best of luck!

[jdempster]

Since upgrading elastic beanstalk to Amazon Linux 2 we've come across the same issue. In the end, I've found the problem is in the ruby aws SDK XML parser, it's configured to strip whitespace. /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.6.0/gems/aws-sdk-core-2.11.546/lib/aws-sdk-core/xml/parser/engines/ox.rb

The solution for me was to create a prebuild hook to patch the ruby file.

#!/bin/bash -eu

if [ -f "/.aws-sdk-core-xml-parser-engines-ox-fix-installed" ]; then
    exit 0
fi

sudo yum install -y patch
patch -u /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.6.0/gems/aws-sdk-core-2.11.546/lib/aws-sdk-core/xml/parser/engines/ox.rb -i .platform/files/aws-sdk-core-xml-parser-engines-ox.patch
touch /.aws-sdk-core-xml-parser-engines-ox-fix-installed
systemctl restart sqsd

--- ox.rb       2021-04-09 20:20:20.862069509 +0000
+++ ox-new.rb   2021-04-09 20:21:05.181225573 +0000
@@ -14,7 +14,7 @@
           Ox.sax_parse(
             @stack, StringIO.new(xml),
             :convert_special => true,
-            :skip => :skip_white
+            :skip => :skip_none
           )
         end

[brad-tilmor]

To piggyback on @jdempster 's answer, I had permission issues using a prebuild hook, so I decided to put it all in an an ebextension file called .ebextensions/fix-xml-parsing.config, if that's helpful to anyone. Not sure yet if this will actually solve the problem, as I can't reliably replicate the error, but I'm hopeful! I'll post an update once enough time has passed that I can be confident it's solved.

UPDATE: This appears to have worked, we haven't run into any new unserialize() errors in the past 2 days. Thanks to @jdempster for this solution!

packages:
  yum:
    patch: []

commands:
  01_apply_patch:
    command: "/tmp/fix-xml-parsing.sh"

files:
  "/tmp/aws-sdk-core-xml-parser-engines-ox.patch":
      mode: "000755"
      content: |
        --- ox.rb       2021-04-09 20:20:20.862069509 +0000
        +++ ox-new.rb   2021-04-09 20:21:05.181225573 +0000
        @@ -14,7 +14,7 @@
                  Ox.sax_parse(
                    @stack, StringIO.new(xml),
                    :convert_special => true,
        -            :skip => :skip_white
        +            :skip => :skip_none
                  )
                end
  "/tmp/fix-xml-parsing.sh":
      mode: "000755"
      content : |
        #!/bin/bash

        if [ -f "/.aws-sdk-core-xml-parser-engines-ox-fix-installed" ]; then
            exit 0
        fi

        if systemctl list-units --full -all | grep -Fq "sqsd.service"; then
            /usr/bin/patch --ignore-whitespace -u /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.6.0/gems/aws-sdk-core-2.11.546/lib/aws-sdk-core/xml/parser/engines/ox.rb -i /tmp/aws-sdk-core-xml-parser-engines-ox.patch
            touch /.aws-sdk-core-xml-parser-engines-ox-fix-installed
            systemctl restart sqsd
            echo "Installed xml parsing patch!"
            exit 0
        fi

        exit 0

[jdempster]

The permission denied issue will most likely be missing the execute flag chmod +x before you zip up the files.

[Nilanth]

@Nilanth No updates at all. We are now moving to docker and ECS to use SQS more directly to go around their daemon that's breaking everything. We have some silly hacky fixes similar to what @JulioPablo did for the time being. technically the base64 encoding of your message would be the best "fix" we had that work also but feels terrible to code this way. best of luck!

@cbcoolbox Which driver are you using for SQS in ECS?

[cbcoolbox]

@Nilanth We have a separate container for jobs and laravel reads directly from SQS. Pretty sure it's just built into laravel (I don't use laravel we are helping other products on our team that do).

[davecalnan]

@jdempster and @brad-tilmor this has worked a treat for us thank you so much for sharing!

[Nilanth]

@jdempster and @brad-tilmor this worked for me too, Thanks for sharing.

[pushpak]

Looks like this issue has been fixed in the latest platform version of 3.3.4

[fylzero]

Marking as stale. If this issue persists as a problem, please open a new issue. Thanks!