Avro + JSON Envelope support (`-s avro` + `-J`) _together_ unsupported when installed via brew

[NJAldwin]

Installing kafkacat from the Homebrew formula provides a version of Kafkacat that supports Avro and also supports JSON.

This is because it is built with --enable-json and --enable-avro: https://github.com/Homebrew/homebrew-core/blob/d5fb4c1209ea1dfdac85d910e8c708d1268a2c6c/Formula/kafkacat.rb#L21-L27

However, when trying to use Avro with a JSON envelope (both features together), the following error appears: This build of kafkacat lacks support for emitting JSON-formatted message keys and values: try without -J or build kafkacat with yajl from https://github.com/edenhill/yajl

$ kafkacat -C -e -J -F kafkacat.production.public.config -r 'https://REDACTED' -s avro -Z -t some-topic
% Reading configuration from file kafkacat.production.public.config
% ERROR: This build of kafkacat lacks support for emitting JSON-formatted message keys and values: try without -J or build kafkacat with yajl from https://github.com/edenhill/yajl

This error does not appear when running via docker, e.g.

$ docker run --rm -it --network host -v"$(pwd)/kafkacat.production.public.config:/root/kafkacat.production.public.config" edenhill/kafkacat:1.6.0 \
  kafkacat -C -e -J -F /root/kafkacat.production.public.config -r 'https://REDACTED' -s avro -Z -t some-topic
% Reading configuration from file /root/kafkacat.production.public.config
{"topic":"some-topic","partition":0,"offset":32579,"tstype":"create","ts":1601482702102,"broker":4,"key":null,"payload":{"abc": "def"}}

I take it from the error message that this is due to the fork of yajl having some necessary feature, while brew installs the upstream yajl.

However, when I look at the fork repo, GitHub claims it's even with master -- suggesting any such changes have been merged upstream already (and both the fork and upstream have been unchanged for 5 years):

If this is the case, what else could be causing this issue?

For what it's worth, if it is necessary to install the yajl fork, I have opened a discussion in Homebrew to try to figure out how that might be added to the kafkacat formula: https://github.com/Homebrew/discussions/discussions/21

For right now, we are able to run kafkacat using docker, as described above. However, it would be ideal if the homebrew version of kafkacat supported all features :)

[edenhill]

Your analysis is correct, kafkacat requires my fork of libyajl to output message key/value JSON as is.

My changes are on the edenhill branch: https://github.com/edenhill/yajl/tree/edenhill

The change is the ability to emit JSON verbatim to avoid having to parse it and then re-emit it, but we should simply do this as a less efficient fallback when my yajl is not available.

[NJAldwin]

Thanks for the quick reply.

The change is the ability to emit JSON verbatim to avoid having to parse it and then re-emit it, but we should simply do this as a less efficient fallback when my yajl is not available.

Should I create a separate issue for the fallback?

My changes are on the edenhill branch

A very small QOL change to start might be to update the error message from build kafkacat with yajl from edenhill/yajl to build kafkacat with the edenhill branch in the edenhill/yajl fork of yajl for clarity?

---

One thing that was suggested in the Homebrew thread I linked above was to upstream the changes in yajl. Any chance the yajl project might be interested in your changes? Sounds like being more efficient about parsing would be beneficial :)

[edenhill]

I did consider upstreaming the change, but it seems that Yajl is an abandoned project, which means we should probably switch to Jansson.

[NJAldwin]

@edenhill does your yajl fork contain any backwards-incompatible functionality? It sounds like (from the thread linked above) one straightforward option here would be to replace the brew yajl with your fork, if the upstream is abandoned.

[DarrenBishop]

Is there a temporary workaround in the meantime?

I'm on macOS, installed via Homebrew. Just tried cloning and building, but it's been a while since I've used configure and make and it's fast becoming a yak-shaving exercise

ps. can't express how awesome kafkacat is

[DarrenBishop]

Ok, figured it out.

For any macOS users that arrive here...

brew uninstall --ignore-dependencies yajl
brew edit yajl

Add the following to the formulae under license (full example formulae linked in this gist)

  head "https://github.com/edenhill/yajl.git",
    branch: "edenhill"

Then run:

brew unlink yajl
brew install --HEAD yajl 
brew install --build-from-source kafkacat

I believe you are now good to go.

[Hubbitus]

Sorry, I also hit such a problem on Linux. Do you plan to address that issue?