secor
secor copied to clipboard
pinterest
Cannot make secor work with AWS instance profile
Hi guys, we are prohibited form using the hardcoded AWS credentials (that we had been using until now) and trying to make secor work with AWS instance profile or role.
I didn't find any documentation but from the secor.common.properties comment we infer that we need to leave the credentials blank and set the role name.
`# AWS authentication credentials. "
Leave empty if using IAM role-based authentication with s3a filesystem.
aws.access.key= aws.secret.key=
Session token only required if using temporary S3 access keys
aws.session.token= aws.role= ` We specified the role name (not sure what to do with aws.session.token) but still get the same error:
2021-02-22 12:55:27,900 [main] (com.pinterest.secor.main.ConsumerMain:87) ERROR Consumer failed java.lang.RuntimeException: Failed to find required configuration option 'aws.access.key'. at com.pinterest.secor.common.SecorConfig.checkProperty(SecorConfig.java:725) at com.pinterest.secor.common.SecorConfig.getString(SecorConfig.java:731) at com.pinterest.secor.common.SecorConfig.getAwsAccessKey(SecorConfig.java:417) at com.pinterest.secor.util.FileUtil.configure(FileUtil.java:75) at com.pinterest.secor.main.ConsumerMain.main(ConsumerMain.java:70)
how to make secor use instance profile or role?
thank you for your help!
David
You want to make sure there is nothing set for aws configs. You might need to trace the code a little bit for that. You can also try between
HadoopS3UploadManager.java and S3UploadManager.java
On Mon, Feb 22, 2021 at 5:04 AM David [email protected] wrote:
Hi guys, we are prohibited form using the hardcoded AWS credentials (that we had been using until now) and trying to make secor work with AWS instance profile or role.
I didn't find any documentation but from the secor.common.properties comment we infer that we need to leave the credentials blank and set the role name.
`# AWS authentication credentials. " Leave empty if using IAM role-based authentication with s3a filesystem.
aws.access.key= aws.secret.key= Session token only required if using temporary S3 access keys
aws.session.token= aws.role= ` We specified the role name (not sure what to do with aws.session.token) but still get the same error:
2021-02-22 12:55:27,900 [main] (com.pinterest.secor.main.ConsumerMain:87) ERROR Consumer failed java.lang.RuntimeException: Failed to find required configuration option 'aws.access.key'. at com.pinterest.secor.common.SecorConfig.checkProperty(SecorConfig.java:725) at com.pinterest.secor.common.SecorConfig.getString(SecorConfig.java:731) at com.pinterest.secor.common.SecorConfig.getAwsAccessKey(SecorConfig.java:417) at com.pinterest.secor.util.FileUtil.configure(FileUtil.java:75) at com.pinterest.secor.main.ConsumerMain.main(ConsumerMain.java:70) how to make secor use instance profile or role? thank you for your help! David
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/1873, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP77F265ADX7NWVKVFFTTAJI5RANCNFSM4YAPPB7A .
Henry, thank you for your advice - switching to s3a and HadoopS3UploadManager did the trick - secor started implicitly using instance profile :)
Glad it worked out. You can also update README on how to setup secor for AWS IAM roles
On Tue, Feb 23, 2021 at 5:22 AM David [email protected] wrote:
Henry, thank you for your advice - switching to s3a and HadoopS3UploadManager did the trick - secor started implicitly using instance profile :)
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/1873#issuecomment-784198805, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP77QCMK7K6X6RWOK6F3TAOTX7ANCNFSM4YAPPB7A .
Henry, I will gladly do update the README file. How do I do it - via pull request or just attach here?
We put the IAM roles authentication config in production yesterday and since then we see duoble the CPU usage by secor. We need twice the amount of nodes (had 3 now need 6 or we are falling behind with data being queued up)
The only thing we changed is authentication. Is it possible the the code checking on re-authentication (AWS token expiration) all the time and causes this CPU drain?
Thank you for your help Henry
You can provide a PR for README file change.
For CPU usage increase, a quick way to verify is usually doing a few thread dumps while the process is running to see where the CPU is spent most time on. Otherwise tools like flame graph or some JVM profiling tool (Yourkit?) can help too.
On Tue, Mar 2, 2021 at 9:13 AM David [email protected] wrote:
Henry, I will gladly do update the README file. How do I do it - via pull request or just attach here?
We put the IAM roles authentication config in production yesterday and since then we see duoble the CPU usage by secor. We need twice the amount of nodes (had 3 now need 6 or we are falling behind with data being queued up)
The only thing we changed is authentication. Is it possible the the code checking on re-authentication (AWS token expiration) all the time and causes this CPU drain?
Thank you for your help Henry
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/1873#issuecomment-789066688, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP7ZQ3KHRF7HEW5YJGDDTBUMCLANCNFSM4YAPPB7A .
Henry, we run into an issue with secor instance profile in AWS China region.
The S3 code that works in regular AWS breaks in AWS China with this error:
com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: SMY7YNMPSYCKSKNJ
I asked for the clarification from AWS Support, and here is their answer:
"This request is sent to Amazon Global region, not China region;
So let us check the region setting first.
When using resources in China region, we must clearly specify the region information.
I saw you use Amazon Java SDK V1, you could use below code to set the region: AmazonS3 s3 = AmazonS3ClientBuilder.standard().withRegion(Regions.CN-NORTH-1).build(); " Would you please check the code, how easy would it be to patch it up?
Thank you for your help! David
created pull request https://github.com/pinterest/secor/pull/2095 with instructions on instance profile setup
I merged your PR, looks like you have at least a workaround.
About code code, the place we are creating S3Client is in:
src/main/java/com/pinterest/secor/uploader/S3UploadManager.java
client = new AmazonS3Client(provider, clientConfiguration);
You can take a look to see whether it's possible to pass in the region information in clientConfiguration or chaining with that withRegion() call
On Wed, Jun 2, 2021 at 3:59 AM David @.***> wrote:
created pull request #2095 https://github.com/pinterest/secor/pull/2095 with instructions on instance profile setup
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/1873#issuecomment-852930628, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP72MJ2HR2W2FEP2A4OLTQYFI7ANCNFSM4YAPPB7A .