charts
charts copied to clipboard
jfrog
[Artifactory HA] 403 Errors using s3-storage-v3 Template
Is this a request for help?:
yes
Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report
Version of Helm and Kubernetes:
| Version | |
|---|---|
| Helm: | v3.2.2+ga6ea66 |
| k8s: | v1.18.9 |
Which chart:
center/jfrog/artifactory-ha
What happened: Using S3 as the binary store does not upload to S3 Object Store. Errors in the log produce the following errors:
[jfrt ] [ERROR] [7b2ddbde9c2d661d] [.c.EventualDownstreamWorker:74] [-cluster-s3-worker-4] - EventualDownstreamWorker caught error while operating on actions for sha1 1f210f9518d447d88bcfe69956ae04a9eee5ef66:
com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: AB689DEB5AC8A6F2; S3 Extended Request ID: RJKkvLmni8KBEhPnbTz9WoE+HFX1hwvJjKC/eDpi63BMzha/cyqAjVmfJf7VUSqlYf38hUCxR4w=; Proxy: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1828)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1412)
What you expected to happen: Files would be copied to S3 Bucket
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know:
The following is my current binarystore.xml file:
<!-- AWS S3 V3 -->
<config version="2">
<chain> <!--template="cluster-s3-storage-v3"-->
<provider id="cache-fs-eventual-s3" type="cache-fs">
<provider id="sharding-cluster-eventual-s3" type="sharding-cluster">
<sub-provider id="eventual-cluster-s3" type="eventual-cluster">
<provider id="retry-s3" type="retry">
<provider id="s3-storage-v3" type="s3-storage-v3"/>
</provider>
</sub-provider>
<dynamic-provider id="remote-s3" type="remote"/>
</provider>
</provider>
</chain>
<provider id="sharding-cluster-eventual-s3" type="sharding-cluster">
<readBehavior>crossNetworkStrategy</readBehavior>
<writeBehavior>crossNetworkStrategy</writeBehavior>
<redundancy>3</redundancy>
<property name="zones" value="local,remote"/>
</provider>
<provider id="remote-s3" type="remote">
<zone>remote</zone>
</provider>
<provider id="eventual-cluster-s3" type="eventual-cluster">
<zone>local</zone>
</provider>
<!-- Set max cache-fs size -->
<provider id="cache-fs-eventual-s3" type="cache-fs">
<maxCacheSize>5e+10</maxCacheSize>
<cacheProviderDir>cache</cacheProviderDir>
</provider>
<provider id="s3-storage-v3" type="s3-storage-v3">
<testConnection>false</testConnection>
<region>us-east-1</region>
<bucketName>example-development-artifactory</bucketName>
<path>artifactory/filestore</path>
<endpoint>https://s3.amazonaws.com</endpoint>
<maxConnections>50</maxConnections>
<useInstanceCredentials>true</useInstanceCredentials>
<refreshCredentials>true</refreshCredentials>
<roleName>arn:aws:iam::XXXXXXXXXXXX:role/artifactory-k8s-s3-role</roleName>
<usePresigning>false</usePresigning>
<signatureExpirySeconds>300</signatureExpirySeconds>
<kmsServerSideEncryptionKeyId>alias/development-artifactory-s3</kmsServerSideEncryptionKeyId>
<property name="s3service.server-side-encryption" value="aws:kms"></property>
</provider>
The above file has been modified many times trying to resolve the 403 errors.
The Helm Chart does not seem to respect setting these values in my passed in values.yml when installing the chart, so I have to manually edit the xml.
Values.yml file:
namespace: artifactory-ha
database:
type: postgresql
driver: org.postgresql.Driver
url: 'jdbc:postgresql://artifactory-aurora-db.development.com:5432/postgres'
user: development_artifactory
password: privo123
artifactory:
masterKey: 7620e1526aae4efc442f0db0e3d93a774a05.......
joinKey: e2a8aa7665225f7c1644f52e0c47e62741e1303...............
persistence:
type: aws-s3-v3
awsS3V3:
region: us-east-1
bucketName: example-development-artifactory <--fake bucket name
endPoint: s3.us-east-1.amazonaws.com
kmsServerSideEncryptionKeyId: development-artifactory-s3
annotations:
iam.amazonaws.com/role: arn:aws:iam::XXXXXXXXXXXX:role/artifactory-k8s-s3-role
node:
replicaCount: 0
postgresql:
enabled: false
I think the behavior here is the pod isn't using the Service Account mapped IAM Role assigned to it, it's using the Node's IAM Role.
My current binarystore.xml has the following (with <useInstanceCredentials> false instead of true):
<provider id="s3-storage-v3" type="s3-storage-v3">
<testConnection>false</testConnection>
<region>us-east-1</region>
<bucketName>example-development-artifactory</bucketName>
<path>artifactory/filestore</path>
<endpoint></endpoint>
<maxConnections>50</maxConnections>
<useInstanceCredentials>false</useInstanceCredentials>
<usePresigning>false</usePresigning>
<signatureExpirySeconds>300</signatureExpirySeconds>
</provider>
and no change to the pod using the worker node IAM Role.
I have the same issue but I can see useInstanceCredentials is on true.
I'm using Kiam for IAM role, and I don't see in the logs the Artifactory using the IAM role at all. I have others services running with Kiam and works fine.
File: binarystore.xml
<!-- AWS S3 V3 -->
<config version="2">
<chain>
<provider id="cache-fs" type="cache-fs">
<provider id="eventual" type="eventual">
<provider id="retry" type="retry">
<provider id="s3-storage-v3" type="s3-storage-v3"/>
</provider>
</provider>
</provider>
</chain>
<!-- Set max cache-fs size -->
<provider id="cache-fs" type="cache-fs">
<maxCacheSize>5e+09</maxCacheSize>
<cacheProviderDir>cache</cacheProviderDir>
</provider>
<provider id="s3-storage-v3" type="s3-storage-v3">
<testConnection>false</testConnection>
<region>eu-west-1</region>
<bucketName>dummy-s3-artifactory</bucketName>
<path>artifactory/filestore</path>
<endpoint></endpoint>
<maxConnections>50</maxConnections>
<useInstanceCredentials>true</useInstanceCredentials>
<usePresigning>false</usePresigning>
<signatureExpirySeconds>300</signatureExpirySeconds>
</provider>
</config>
Helm values
artifactory:
...
persistence:
enabled: false
type: aws-s3-v3
awsS3V3:
region: eu-west-1
bucketName: dummy-s3-artifactory
annotations:
iam.amazonaws.com/role: "arn:aws:iam::xxxx:role/role-with-permissions-for-s3-dummy-s3-artifactory"
...
Role arn:aws:iam::xxxx:role/role-with-permissions-for-s3-dummy-s3-artifactory
{
"Statement": [
{
"Action": [
"s3:*" <--- just for testing
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::dummy-s3-artifactory",
"arn:aws:s3:::dummy-s3-artifactory/*"
]
}
],
"Version": "2012-10-17"
}
I followed the documentation from here https://artifacthub.io/packages/helm/jfrog/artifactory#aws-s3-v3
@chukka I'm still having this issue.. I've followed this kb article, but still getting a 403 error.
2022-01-11T16:45:02.182Z [jfrt ] [ERROR] [55cd7f27bbdd45b5] [ctoryContextConfigListener:128] [art-init ] - Application could not be initialized: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: REDACTED; S3 Extended Request ID: REDACTED; Proxy: null)
java.lang.reflect.InvocationTargetException: null
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.artifactory.lifecycle.webapp.servlet.ArtifactoryContextConfigListener.configure(ArtifactoryContextConfigListener.java:271)
at org.artifactory.lifecycle.webapp.servlet.ArtifactoryContextConfigListener$1.run(ArtifactoryContextConfigListener.java:124)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'nuGetRepositoryTypeHelper': Unsatisfied dependency expressed through field 'repoService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'repositoryServiceImpl': Unsatisfied dependency expressed through field 'aclService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'securityServiceImpl': Unsatisfied dependency expressed through method 'setBuildService' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'buildServiceImpl': Unsatisfied dependency expressed through method 'setProjectsService' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'projectsServiceImpl': Unsatisfied dependency expressed through method 'setStorageService' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'storageServiceImpl': Unsatisfied dependency expressed through method 'setBinaryStore' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'binaryServiceImpl': Invocation of init method failed; nested exception is java.lang.RuntimeException: Failed to test connection withS3AwsBinaryProvider, container:artifactory-filestore-xxx
Verifying that the pod has the environment values:
bash-4.4$ env |grep AWS
AWS_DEFAULT_REGION=us-east-1
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_REGION=us-east-1
AWS_ROLE_ARN=arn:aws:iam::XXXXXX:role/artifactory-s3-role
I've followed this AWS article, and was able to create a pod that can invoke the aws s3 commands with the "serviceaccount"
@privomark @Loucool111 @stevep-arm - can you please reach out to jfrog support team who can do a one on one session to diagnose and fix the issue with your values file?
Something to check if you encounter this issue: at some point in the history the chart changes the name of the service account from artifactory-artifactory to artifactory. Make sure your role's policy permits this new serviceaccount name.
