Henry Haiying Cai

You want to make sure there is nothing set for aws configs. You might need to trace the code a little bit for that. You can also try between HadoopS3UploadManager.java...

Glad it worked out. You can also update README on how to setup secor for AWS IAM roles On Tue, Feb 23, 2021 at 5:22 AM David wrote: > Henry,...

You can provide a PR for README file change. For CPU usage increase, a quick way to verify is usually doing a few thread dumps while the process is running...

I merged your PR, looks like you have at least a workaround. About code code, the place we are creating S3Client is in: src/main/java/com/pinterest/secor/uploader/S3UploadManager.java client = new AmazonS3Client(provider, clientConfiguration); You...

I think it would be easier if you add a few print statement in DateMessageParser class to see what was going on. On Wed, Dec 30, 2020 at 5:47 AM...

This is expected. You most likely have messages with timestamps jumping back and forth between old and new dates. In that case multiple data files will be created (one for...

Not at the moment, it's probably not too hard to add an enhancement of using field name lookup. On Fri, Jul 23, 2021 at 12:25 AM shantam04 ***@***.***> wrote: >...

Exact-once upload has not much to do with which upload manager, it depends on the following things: 1. The file name generation is deterministic: kafka-partition-number + begin-kafka-offset 2. S3 upload...

There is no get, it's a file replacing operation on S3. On Mon, Oct 19, 2020 at 11:31 AM Jay Patel wrote: > Thanks Henry, does any of this operation...

The parquet support code was checked in a while ago by a community member, you can check the history of this file check in to find the original author. I...