kanadi
kanadi copied to clipboard
zalando-nakadi
Adding Avro support capability
This PR is part of the effort to introduce Avro capabilities to Nakadi Clients.
This PR contains modifications that introduces Avro publishing and consumption capability, kept in mind to introduce no changes to existing users of the client.
@samizzy Thank you for the PR.
It looks the build is failing. could you please check it?
and it might be worth to check the DCO section that required the commits to be signed off.
@samizzy forgot to post a comment the review: besides of the code changes, the documentation update is needed to make it clear how to use the new capabilities.
btw, it seems that failing tests are related to this issue: https://github.com/adyach/nakadi-docker/issues/5 as it was mentioned in the other PR.
@samizzy given changes to the license https://github.com/akka/akka/pull/31561, no new features will be added here.
Instead propose to add this either to update the existing library for ZIO or create a new one and implement avro there.
but the main issue with Nakadi testing remains: we need to resolve this issue: https://github.com/adyach/nakadi-docker/issues/5
@gchudnov @vadeg I have just added a merge commit onto @samizzy PR which brings the latest changes from kanadi master (including migration from Akka to Pekko) so its in a state where it can be worked on.
One additional thing that I had to add to the merge commit is having to manually convert between CharSequence and String. Not sure why this is the case (compiling with JDK 11) but I can look into it later.
If this gets merged it should be noted that unlike the JSON format, the Avro parsing isn't streamed (should probably mention this in README.md). This is possible to do via jsurfer, i.e. https://github.com/wanglingsong/JsonSurfer#binaray-format-jackson-only and you can use Pekko's existing JsonFraming as inspiration how to create a Flow that properly streams Avro binary data
So CI failed
[error] org.specs2.specification.core.FatalExecution: Error from server, response is Problem(None,Bad Request,400,Some(Illegal enum value: 'avro_schema'. Possible values: [json_schema] (through reference chain: org.zalando.nakadi.domain.EventTypeBase["schema"]->org.zalando.nakadi.domain.EventTypeSchemaBase["type"])),None) (package.scala:83) [error] org.zalando.kanadi.api.package$.$anonfun$processNotSuccessful$1(package.scala:83) [error] CAUSED BY [error] org.zalando.kanadi.models.GeneralError: Error from server, response is Problem(None,Bad Request,400,Some(Illegal enum value: 'avro_schema'. Possible values: [json_schema] (through reference chain: org.zalando.nakadi.domain.EventTypeBase["schema"]->org.zalando.nakadi.domain.EventTypeSchemaBase["type"])),None) (package.scala:83) [error] org.zalando.kanadi.api.package$.$anonfun$processNotSuccessful$1(package.scala:83)
I guess this is due to the nakadi docker image used in CI is an outdated one which doesn't support avro
@mdedetrich Thank you for the updates! no worries, it will take some time to polish and merge this PR