Henry Haiying Cai

I think it would be nice to have this PR merged if you already used in your production environment. I think you just need to resolve the conflicts with the...

Do you want to take a look at these two PRs? https://github.com/pinterest/secor/pull/1449 https://github.com/pinterest/secor/pull/1443 On Sun, Nov 1, 2020 at 1:07 PM David wrote: > We run a two fairly large...

Do you see this line in your log file? LOG.info("Shut down hook with priority {} added to shut down hook registry", priority); On Wed, Nov 4, 2020 at 12:27 AM...

I think we can remove the folder if it's more than X days old and you can configure that X depending on whether your late arrivals will be X days...

Richard, If you need to set those two params separately, you can modify the secor code to introduce two separate params. On Mon, Nov 30, 2020 at 4:14 AM Richard...

I don't think it will hurt to introduce that parameter, feel free to submit a PR. On Tue, Dec 1, 2020 at 1:01 AM Richard Grossman wrote: > Hi >...

Your issue seems related to this upstream bug: https://issues.apache.org/jira/browse/PARQUET-1684, I am not sure whether there will be a short-term solution. On the other hand, when you declare the hive table...

Do you see consumer group rebalance during those times? There should be some logging messages indicating the rebalance was happening, that usually is a period of time which might cause...

You can do a few thread dump to see what the process stuck at. Adding memory probably will help parquet conversion. On Tue, Jul 27, 2021 at 1:52 AM aksamit26...

I think by default most of the output format will only handle the value portion. If you want the key portion, you might need to overwrite the AvroParquetFileReaderWriterFactory to take...