Praveen

@levin81 - S3N is stable right now and am testing S3A now. Will have some updates soon for S3A.

@levin81 Added a comment to https://github.com/qubole/streamx/issues/30 detailing the s3a issues there.

Hi @zzbennett Sorry to respond late. Yes, we have found multiple issues with S3A (thread leak and httpClient related issues). So far, the experience with using NativeS3FileSystem is very stable....

@zzbennett You are right. They are not going to add the Roles (temp creds) support in S3N and S3A is the way forward. I will look into this issue and...

Yeah right, you need to remove the keys (or it wont use roles and the keys are invalid). I will add a note for this.

@zzbennett Please look at https://github.com/qubole/streamx/issues/30 for issues related to S3A.

Hi @dhamilton Thanks for reporting it. This seems to be caused due to a bug in s3a where it does not clean up threads, and instead goes on creating worker...

@dhamilton Can you try it out now. I have pushed a commit which upgrades the AWS SDK (and hadoop), and this should fix this thread leak issue. Please let us...

This usually points to some wrong config related to s3a, check the access/secret keys and the file system impl configs. Task will try creating S3Storage and it fails when there...

@dhamilton I have been testing this and hitting number of issues with S3A. In my testing, I dont see any NativeS3FileSystem issues, so I suggest you to use that instead....