New version 0.27 does not properly handle chunked encoding when use_compression is enabled

[dolartoday]

Version 0.27 now works correctly with chunked encoding, however, if the use_compression option is enabled it fails.

[RonKass]

indeed. When doing some debugging, it seems it inflates the json chunks, which come out as partial json code, rather than the complete, which then fails to be json-decoded.

A printout of an example {"created_at":"Tue Jan 26 14:08:26 +0000 2016","id":691986054312648705,"id_str":"691986054312648705","text":"\u4e2d\u9014\u534a

which then throws the following error upon decoding: ERROR :: unexpected end of string while parsing JSON string, at character offset 127 (before "(end of string)") at /usr/local/share/perl5/AnyEvent/Twitter/Stream.pm line 127.

[RonKass]

Ok.. I've fixed it. Basically, the problem was that chunks come in with partial blocks of the JSON. And the code just assumed that each chunk is a complete json block, rather than concatenating the inflated text and sending over complete json blocks. So... I am no expert in writing patches.. but here's my attempt.

--- /usr/local/share/perl5/AnyEvent/Twitter/Stream.pm   2014-04-18 05:08:53.000000000 +0000
+++ /usr/local/share/perl5/AnyEvent/Twitter/Stream.pm.new       2016-01-26 17:09:58.461859860 +0000
@@ -53,6 +53,8 @@
     my $on_event        = delete $args{on_event};
     my $timeout         = delete $args{timeout};

+    my $message_accumulated = "";
+
     my $decode_json;
     unless (delete $args{no_decode_json}) {
         require JSON;
@@ -191,7 +193,13 @@
                                $_zstatus = $zlib->inflate(\$input, \$message);
                                return unless $_zstatus == Z_OK || $_zstatus == Z_BUF_ERROR;
                                } while ( $_zstatus == Z_OK && length $input );
-                               $on_json_message->($message);
+
+                               $message_accumulated .= $message;
+                               if ($message =~ /\n$/) {
+                                   my @messages = split('\n',$message_accumulated);
+                                   foreach (@messages)
+                                       {$on_json_message->($_); }
+                                   $message_accumulated = "";
+                               }
                         } else {
                             die "Don't know how to decode $headers->{'content-encoding'}"
                         }

Basically, the patch creates a

my $message_accumulated = "";

which accumulates the deflated json code (parts) and once a \n is detected in the accumulated message, it splits it and sends it for processing...

                               $message_accumulated .= $message;
                               if ($message =~ /\n$/) {
                                   my @messages = split('\n',$message_accumulated);
                                   foreach (@messages)
                                       {$on_json_message->($_);}
                                   $message_accumulated = "";
                               }

..instead of the line that sends all chunks right away

                               $on_json_message->($message);

[frioux]

I can confirm that this issue exists and for people too lazy to hand patch the code disabling compression works.