Is there any alternative to AvroParquetWriter so that we can avoid using IndexedRecord?

[shantam04]

We are using Secor to write data from Avro records Kafka to Parquet records in Azure.

We use AvroParquetWriter to convert deserialized Avor records to Parquet, but as AvroParquetWriter works with IndexedRecord interface - schema evolution is very difficult as it matches on the index of record fields with schema. Any out-of-order schema evolution leads to backward compatibility issues when converting to Parquet.

[HenryCaiHaiying]

Not at the moment, it's probably not too hard to add an enhancement of using field name lookup.

On Fri, Jul 23, 2021 at 12:25 AM shantam04 @.***> wrote:

We are using Secor to write data from Avro records Kafka to Parquet records in Azure.

We use AvroParquetWriter to convert deserialized Avor records to Parquet, but as AvroParquetWriter works with IndexedRecord interface - schema evolution is very difficult as it matches on the index of record fields with schema. Any out-of-order schema evolution leads to backward compatibility issues when converting to Parquet.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/2132, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP72QLA25XCE7B7SKTX3TZEKQFANCNFSM5A3O6WWQ .