Deserialize messages in smile format

[rrva]

Autodetect and deserialize messages in smile format

https://github.com/FasterXML/smile-format-specification

[CLAassistant]

All committers have signed the CLA.

[emaxerrno]

@weeco - let's think how to expose this, maybe as a plugin framework? (like RPK plugins?) - so that way people can plugin custom serializations.

[rrva]

@senior7515 oh please noo this delays this even further, exactly what https://github.com/provectus/kafka-ui chose to answer when I made a PR there ;)

The existing deserialization support is non-pluggable and until that gets rewritten into plugins, I would have to run a fork of kowl which is tedious to rebase when fixes arrive... So if plugins are a while away I argue for this to be included in the meantime, it is not hard to extract once that support is there. If you look at the patch it is small and adds just one other case to the existing deserialization cases. Other than that I fully agree with the architectural argument of having plugins.

[rrva]

@weeco please reconsider this... has something happened on the plugin architecture which make this easy to add (how?) or could it be merged as-is?

[weeco]

Hey @rrva , we haven't worked on the plugin architecture yet.

However, I think it won't hurt to add this by default, even though I'd like to preserve the right to remove it (or put behind a feature flag / plugin) in an upcoming version, if this conflicts with the other deserialization methods. I think this format is rarely used in the Kafka community and is an ideal usecase for such a plugin.

[rrva]

thanks a lot! I understand your reservations. I'll try it and report back if there are any issues with it. I do not see it hurt the existing deserializer path a lot for performance etc.