Discussion about Schema Evolution

[giampaolotrapasso]

Not really an issue :)

Are you planning to investigate how to evolve the schema with the current serialization format? Official documentation does some examples but I did not found something more extended.

What I mean is showing how to manage the schema for these cases:

• adding a new field
• renaming a field
• removing a field
• splitting an event in two

using always the same serialization format. I'm not interested in generic ways to handle the problem, instead into specific examples to show how much work is needed with every serialization format.

About generic evolution, there's an interesting project about schema evolution. But it seems at an early stage ATM.

[dnvriend]

Like all the '(A small) study project on' projects on my github page, including the persistence-plugins btw :) are just that, study projects how to get things done in Scala and how to use the supporting frameworks/utility libraries. I find them valuable as I use them regularly on my job and point to my colleagues how to get things done. It's a growing projects and personally, I'm really interested in schema evolution. For some reason, I really like Protobuf and want to know a whole lot more about it. It comes out of the box with Akka, so why not! The cases you mentioned is really what I want to examine, also the use cases on the persistence-schema-evolution page.

[giampaolotrapasso]

Ok, if you arrange some examples with protobuf, I'll try to mimic them using avro

[dnvriend]

That would Be great!! 👍👍

[calvinlfer]

I found this helpful: https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html

20 minute mark has useful information on Avro's schema evolution https://www.youtube.com/watch?v=GfJZ7duV_MM

Also Google's documentation provides some good guidelines for Schema Evolution for Protocol Buffers: https://developers.google.com/protocol-buffers/docs/javatutorial#why-use-protocol-buffers

[dnvriend]

Thanks for the information!

[rrodseth]

Any progress on this? I was hoping to find a follow-up to http://giampaolotrapasso.com/akka-persistence-using-avro-serialization/

Thanks!

[dnvriend]

Hi, its a great blog by @giampaolotrapasso. I don't know if there will be an update to the blog or the project.

[giampaolotrapasso]

Thanks @rrodseth for you interest. I thought never one read that :). Give me some days to recap and go on.

[rrodseth]

Thanks for the responses. Any guidance would be much appreciated. I thought it made sense to use Avro for akka-persistence since that's what Kafka recommends, but the Akka docs are all very much Protobuf-focussed.