erlavro icon indicating copy to clipboard operation
erlavro copied to clipboard

backwards and forward schema evolution

Open sudeepdino008 opened this issue 1 year ago • 0 comments

I was trying out erlavro for schema evolution. I'm not sure what the stance on evolution is here. Is it not supported? Or is there a way to support it that I'm not aware of. For example:

the schema (block-ethereum.avsc) has a lot of fields deleted (in contrast to the data provided which was encoded using an earlier version). No additional fields are added or modification done on any of the retained fields in this new schema:

{
   "type":"record",
   "namespace":"com.covalenthq.brp.avro",
   "name":"ReplicationSegment",
   "fields":[
      {
         "name":"startBlock",
         "type":"long"
      },
      {
         "name":"endBlock",
         "type":"long"
      },
      {
         "name":"elements",
         "type":"long"
      },
      {
         "name":"codecVersion",
         "type":"double",
         "default":0.33
      }
   ]
}

the encoded binary is here

when i try running the decoder:

iex> {:ok, scheman} = Avrora.Schema.Name.parse("block-ethereum")

iex> scheman1 = %Avrora.Schema{full_name: scheman.name}

iex> {:ok, schema} = Avrora.Resolver.resolve(scheman1.full_name)

iex> :avro_binary_decoder.decode(specimen, schema.full_name, schema.lookup_table, Avrora.AvroDecoderOptions.options())

** (MatchError) no match of right hand side value: {%{"codecVersion" => 4.3256427312130535e-37, "elements" => 24, "endBlock" => 66, "startBlock" => 1}, <<97, 100, 54, 99, 57, 57, 54, 51, 97, 56, 55, 52, 56, 48, 54, 97, 51, 102, 50, 51, 51, 49, 49, 50, 50, 99, 56, 50, 99, 54, 50, 55, 48, 49, 98, 55, 54, 48, 55, 50, 98, 48, 100, 55, 55, 50, 49, 56, ...>>}
    (erlavro 2.9.8) /Users/sudeep/repos/rudder/deps/erlavro/src/avro_binary_decoder.erl:66: :avro_binary_decoder.decode/4
    iex:6: (file)

How can i support backwards of forward evolution of schema here?

sudeepdino008 avatar Apr 18 '23 05:04 sudeepdino008