ParquetWriter write method is taking too long to write single avro message

[aksamit26]

When i am reading data form Kafka , i am deserialising it to Avro record. But ParquetWriter write method is taking around 2~5 secs to write one single avro record. Is there any such issue surfaced earlier. What can be possible solve for this ?

[HenryCaiHaiying]

You can do a few thread dump to see what the process stuck at. Adding memory probably will help parquet conversion.

On Tue, Jul 27, 2021 at 1:52 AM aksamit26 @.***> wrote:

When i am reading data form Kafka , i am deserialising it to Avro record. But ParquetWriter write method is taking around 2~5 secs to write one single avro record. Is there any such issue surfaced earlier. What can be possible solve for this ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/2133, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP74TKFG6EMN5TRTVTJDTZZXVVANCNFSM5BBYSN7A .