Activity is not JSON serializable

[vasanthsai]

Hi,

Over View of Issue: In our django APP celery uses JSON serialization instead of pickle for security reasons, where as feedly(Stream-Framework) assumes pickle serialization.

Issue: When we add a activity to user basic feed with few followers list the following error is showing up EncodeError: Activity(won) 1 5 is not JSON serializable

Attempted Solution: We thought the problem was with time object and instead of using default serializer, i implemented my own serializer while sub classing RedisFeed and RedisAggregatorFeed, which converts time to string while dumping and string to time while loading. But the problem seems to be in a much higher level since activity object itself is being serialized.

Can you Please suggest me a solution without having to change celery configuration.

[tschellenbach]

Going to ask @tbarbugli to pitch in here. We're actually also looking to move away from Pickle.

So for pickle serialization you have the hooks in python which allow you to customize serialization. For JSON however you need to change serializer class.

See for instance: http://stackoverflow.com/questions/3768895/python-how-to-make-a-class-json-serializable You can find more examples when searching for how to handle datetime for instance.

Alternatively we could change the task signature to send a dictionary and not the activity object. It would need a small compatibility layer, but that's not too much work.

@tbarbugli what do you think?

[pinda]

@vasanthsai Do you already have a solution for this? Tried overriding the serializer but could not get the json serializable.

[tvanesse]

@tschellenbach Indeed that would be a really great feature. For the purpose of my application I'm writing a custom JSON serializer for Activity because my feed is sent through a REST API to the front-end, but that would have been nice to have it built-in Stream-Framework.

[tbarbugli]

yes, we are planning to move away from pickle serialisation. I am going to see how soon we can fix this issue and update this conversation.

[meleksomai]

Any comment on this feature @tbarbugli. We are also worried about the implications of using pickle and would be great if it would be possible to use JSON serialization with celery.

[jquacinella]

+1 for this as well. Celery yells at me about using pickle :)

[jquacinella]

@tvanesse How did you go about doing this? IT seems that I can create my own custom JSON serialization for my custom Activity class, but now I seem to get errors stating that my custom Feed class is not JSON serializable. While debugging, I notice it is trying to serialize the Feed class itself, not an object of the Feed class which I find every odd. Can you describe how you fully implemented JSON serialization for celery?

[tvanesse]

@jquacinella It's been a while I did not go into this part of the code. I think I did not change the way activities are serialized for Celery. I just implemented a method on my Manager named get_enriched_activities(self, activities) that takes a list of Activity objects and returns a detailed dictionary with values retrieved from the db. This dict is then sent as a JSON object over HTTP in my API.

[ghost]

Have anyone ever found a solution for this?

[jquacinella]

I ended up doing something similar to tvanesse and handle json serialization on my own; I think I ended up storing a regular python dict in the extra_activities and used that

[ghost]

Yeah, that's what I did too. I changed extra_context to use custom user types instead of bytes. I also stored regular python dicts for those too.

[chanphillip]

Any solution yet? Saw a few posts but couldn't get it resolved for my django project. I tried implementing my own activity serializer, but it gave the same error.

[vlafranca]

Hi, I just found a solution using django rest framework, here it is :

from rest_framework import serializers

class GetActivities(generics.GenericAPIView):
    queryset = ''
    def get(self, request, version):
        feed = manager.get_user_feed(1)
        activities = list(feed[:25])
        return Response(CommentSerializer(activities, many=True).data)

class VerbSerializer(serializers.Serializer):
    id = serializers.ReadOnlyField()
    infinitive = serializers.ReadOnlyField()
    past_tense = serializers.ReadOnlyField()

class CommentSerializer(serializers.Serializer):
    actor_id = serializers.ReadOnlyField()
    target_id = serializers.ReadOnlyField()
    object_id = serializers.ReadOnlyField()
    verb = VerbSerializer() 
    extra_context = serializers.ReadOnlyField()```