schema-registry-transfer-smt icon indicating copy to clipboard operation
schema-registry-transfer-smt copied to clipboard

Published 20 hours ago • verified publisher icon OneCricketeer

Mm2 updates

Open iturcino opened this issue 5 years ago • 8 comments
trafficstars

Fixes OneCricketeer/schema-registry-transfer-smt#25

iturcino avatar Jun 04 '20 13:06 iturcino

Thank you!

Overall makes sense. Few comments.

Otherwise, I'm not sure I understand why it was ultimately needed... MM2 should work on its own since this SMT doesn't package anything but Avro and the Registry client, which the HTTP client hasn't changed that much, and it's shaded here too

OneCricketeer avatar Jun 04 '20 18:06 OneCricketeer

Updated PR according to comments here

iturcino avatar Jun 07 '20 11:06 iturcino

Version bumped to v.0.2.1

iturcino avatar Jun 09 '20 07:06 iturcino

Given that this PR has been approved, can @OneCricketeer please merge it and build new 0.2.1 version?

kokachev avatar Jan 20 '21 06:01 kokachev

are you going to merge that?

jipipayo avatar Mar 10 '21 15:03 jipipayo

Hi all, I no longer use Kafka for my role, nor this SMT, so am not actively looking at maintaining it.

That being said, Confluent has documentation about migrating schemas, and I am curious if anyone has tried those methods and has feedback on how they differ from this repo?

OneCricketeer avatar Jan 17 '23 22:01 OneCricketeer

Hi all, I no longer use Kafka for my role, nor this SMT, so am not actively looking at maintaining it.

That being said, Confluent has documentation about migrating schemas, and I am curious if anyone has tried those methods and has feedback on how they differ from this repo?

The approach listed here involves setting the schema registry into a special mode where you can specify the id of the schema when you load it. This does work and you can use mirror maker without this SMT and then some other program to simply push the existing schema and id's in

Limitations

  • only works if the target schema registry does not contain any of the id's in your source schema registry
  • other schema registries don't support this import mode such as apicurio though you presumably could use their backup and restore process
  • still requires something to clone the schema registry - may even need a smt if your using mm with an evolving schema

I like your approach and think it's odd that it's not part of mirror maker

Limitations of this approach

  • the magic byte isn't always a guarentee that a binary topic is avro such as the internal ksql or __consumer_offsets

my quick fix is to add an ignore list: https://github.com/iturcino/schema-registry-transfer-smt/pull/2

I'm tempted to think it would be better for my use case if it simply ignored the not found or bad magic number

msillence avatar Feb 06 '24 09:02 msillence

anyone can review and merge it?

vl-kp avatar Apr 08 '24 01:04 vl-kp