Consider providing serialization-aware consumer API

[michaelklishin]

Many scenarios in message consumer applications are very common. For example, JSON or MsgPack used for serialization. We should investigate a convenient way to plug common serialization formats into consumer API. Then payload attribute can be a data structure instead of a bag of bytes.

[carlhoerberg]

yes, and clojure data structure (de)serialization:

(defn serialize [data]
  (binding [*print-dup* true] (pr-str data)))

(defn deserialize [str]
  (with-in-str str (read))

suggested content-type: application/clojure

[carlhoerberg]

https://github.com/mefesto/wabbitmq has generic serialization support, could be worth checking out..

[michaelklishin]

We probably can take a page from Welle which in turn steals this idea from Sumo. I don't want this to replace langohr.consumers/subscribe, though, just be an useful option to choose from.

[carlhoerberg]

maybe even include content-encoding, eg. automatic gzip (de)compression of payloads..

[mlimotte]

I've written my own abstraction on top that uses a ring style to attach additional behaviors like this. So I have a wrap-decode-payload wrapper that support JSON or EDN based on content-type. Similarly, I have a wrap-encode-payload for going the other way and several other wrappers for other purposes. Gzip could be supported the same way. This pattern seems to work pretty well.

[michaelklishin]

@mlimotte yes, I was contemplating doing just that.

[danielsz]

Has the release of transit any incidence over this discussion?

[michaelklishin]

Transit doesn't change anything for us. We can't support a single transport only.

[danielsz]

OK. Thanks.

Just to be clear: is it up to everyone to build their abstractions on top of langohr for serialization/deserialization, or are there features/best practices/examples that can help when starting out?

[michaelklishin]

Well, if you can deserialize a byte array in your consumer, you are all set today.

We can provide a protocol and a deserializing consumer function that will do it for you and call your handler with a Clojure data structure, that's pretty much it.

[danielsz]

Thanks, Michael, I'll take the first suggestion for now.

Coming from Ruby, I'm always surprised when there's no native data structures straight off the box. Not a big deal, but it's a philosophy thing that puzzles me.