Fix for Pickle/Redis encoding issue (related to #57)

[izhan]

Thanks again for creating an awesome framework. Really loving it so far.

I wanted to bring attention to a bug I (and other users...see #57) encountered with Redis and extra_context, and our fix for it. I am currently using a RedisNotificationFeed and adding activities with extra_context set to arbitrary input (in this case, user-inputted comments). The script is currently crashing with certain production data, specifically non-ASCII characters in positions 128 to 255. For instance, if extra_context contains the degree symbol °, it throws the error. However, extra_context with 狗 doesn't.

Here's the specific error output:

  File "directory/project/notifications/notification_manager.py", line 49, in do_something
    feed.add(activity)
  File "directory/venv/lib/python2.7/site-packages/stream_framework/feeds/base.py", line 197, in add
    return self.add_many([activity], *args, **kwargs)
  File "directory/venv/lib/python2.7/site-packages/stream_framework/feeds/aggregated_feed/notification_feed.py", line 68, in add_many
    self, activities, **kwargs)
  File "directory/venv/lib/python2.7/site-packages/stream_framework/feeds/aggregated_feed/base.py", line 90, in add_many
    current_activities = self[:self.merge_max_length]
  File "directory/venv/lib/python2.7/site-packages/stream_framework/feeds/base.py", line 305, in __getitem__
    start, bound)
  File "directory/venv/lib/python2.7/site-packages/stream_framework/feeds/base.py", line 347, in get_activity_slice
    ordering_args=self._ordering_args)
  File "directory/venv/lib/python2.7/site-packages/stream_framework/storage/base.py", line 311, in get_slice
    key, start, stop, filter_kwargs=filter_kwargs, ordering_args=ordering_args)
  File "directory/venv/lib/python2.7/site-packages/stream_framework/storage/redis/timeline_storage.py", line 83, in get_slice_from_storage
    key_score_pairs = cache.get_results(start, stop, **result_kwargs)
  File "directory/venv/lib/python2.7/site-packages/stream_framework/storage/redis/structures/sorted_set.py", line 182, in get_results
    key, start=start, num=limit, withscores=True, min=min_score, max=max_score)
  File "directory/venv/lib/python2.7/site-packages/redis/client.py", line 1761, in zrevrangebyscore
    return self.execute_command(*pieces, **options)
  File "directory/venv/lib/python2.7/site-packages/redis/client.py", line 565, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "directory/venv/lib/python2.7/site-packages/redis/client.py", line 577, in parse_response
    response = connection.read_response()
  File "directory/venv/lib/python2.7/site-packages/redis/connection.py", line 569, in read_response
    response = self._parser.read_response()
  File "directory/venv/lib/python2.7/site-packages/redis/connection.py", line 264, in read_response
    response = [self.read_response() for i in xrange(length)]
  File "directory/venv/lib/python2.7/site-packages/redis/connection.py", line 266, in read_response
    response = response.decode(self.encoding)
  File "directory/venv/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xb0 in position 360: invalid start byte

Note that we are running Python 2. The core issue is that pickle.dumps returns bytes that Redis tries to decode to utf-8. Our current solution is disabling this decoding by changing decode_responses to False in stream_framework/storage/redis/connection.py (or removing it completely) i.e.

pool = redis.ConnectionPool(
    host=config['host'],
    port=config['port'],
    password=config.get('password'),
    db=config['db'],
    decode_responses=False
)

What are your thoughts on a long-term solution that works for both Python 2 and 3? I'd be happy to investigate more.

[Atorich]

+1

So activity is encoded in latin1 at serializing via pickle but decodes from utf8 (by default) when load data from storage.