feat: add rest serde

[wassim-k]

First of all, thanks for the great work you've done on this console. We've started using it recently and love it!

We have a use case where IoT devices publish their data to MQTT using a MessagePack format and then that gets forwaded to a topic. But the format is quite complex with custom MessagePack formatters and resolvers implemented in C# where OOB MsgPack serde just doesn't cut it.

So I've created this PR as a proposal for introducing REST based deserialization. It does so by calling a REST API endpoint and passing the message's binary data in the body and expects a JSON response in return. This is quite useful for supporting systems with unconventional data formats or language sepecific proprietary serialization.

This is my first foray into Go land, so please don't hesitate to provide feedback, suggestions and request changes.

If the concept of this PR is acceptable to you, then I'd add some more unit/integration tests around the new serde.

Keen to hear your thoughts, thanks.

[CLAassistant]

All committers have signed the CLA.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[weeco]

Hey @wassim-k , we discussed this PR internally. We believe this is not the best approach to your problem, for two reasons:

1. Console supports a search mode where you can search an entire topic using a push-down filter (Javascript code evaluated inside a VM running i nthe backend). This is more performant than one might assume, Roughly 50k messages can be processed per second per core, depending on the filter and message of course. We believe such a REST serde would be much more inefficient and basically a dealbreaker for the pushdown filters.

2. Sending binary payloads via HTTP is network intensive and for some it may be concerning if Console sends your data to some external HTTP endpoint for different reasons.

We believe the better solution would be either a plugin system where people can write their own custom plugins for deserializing messages or similarly utilizing WASM so that people can write their own deserializer using WASM transforms as well. What do you think?

[wassim-k]

Hi @weeco, thanks for the taking the time to consider my proposal.

Let me explain our setup a little more: Btoh redpanda-console and the service consuming + deserializing the topic are within the same k8s cluster. The service is only accessible locally within the cluster and has a network policy allowing only redpanda-console to access it via http, this has the following advantages:

• Network latency is miniaml since they're both within the same cluster. In my experience there is no noticeable delay when consuming the last 500 message but I do agree that pushdown filters are a lot slower with REST serde but this was a compromise I was accepting when enabling REST for the topic.
• There is no risk of sending network requests exeternally since the service is only accessible locally within the same cluster and there are network policies governing http access.
• The same service that consumes the topic has the most up to date message pack models and deserialization logic that is also exposed as a REST endpoint which means any deployments to the service that include changes to deserialization or models are automatically availabe to the endpoint and subsequently redpanda-console (which is fairly frequent)

But I understand that there are risks involved in enabling such functionality that users might accedintally expose themselves or degrade performance noticeably, so a plugin architecture does sound like a better option going forward. For me personally, I might attempt to implement REST functionality within a plugin for the advantages above, since trying to keep the plugin up to date with logic+model changes might require changes to the deployment pipeline and introduces a lot of complexity compared to a simpe REST endpoint.

Would you be open to supporting REST serde until such plugin architecture is available or do you believe the risks are too high or the use case is too specific to maintain the code? If so, I completely understand and you may close the issue, Thanks again :)

[weeco]

Currently I assume this is very specific to your use-case as we haven't heard about similar requests. I'd recommend you to keep running a forked version of Console for this. I think it's rather unlikely that you'll face conflicts when syncing your fork with origin so it should be relatively low maintenance as well if this works for you.

Thanks again for your efforts and understanding our choice. If we get to work on a plugin-system for custom SerDes I'd be happy to partner with you to make sure we can cover your usecase as well.