charts Unknown blob when pushing to Artifactory , with s3 as the storage

Is this a request for help?: Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes: 3.12.3 and V1.29

Which chart: Artifactory-107.90.8

Which product license (Enterprise/Pro/oss): Pro

JFrog support reference (if already raised with support team): N/A

What happened: Pushing images to our artifactory instance, throws the error unknown blob after pushing images

the error in the logs show


Deploy to 'docker-local:docker-local/mongo/_uploads/6d562ef3-cab2-4b8d-b60e-f4d5b6a05c2d' Content-Length: 0 (estimation) artificial: false

Failed while saving temp file /opt/jfrog/artifactory/var/data/artifactory/filestore/_pre/dbRecord870694586599854700-35d5a484b17b6f0a-docker-local.bin
org.jfrog.storage.binstore.ifc.ClientInputStreamException: Failed to read stream: Failed to read stream: java.io.EOFException
Caused by: org.jfrog.storage.binstore.ifc.ClientInputStreamException: Failed to read stream: java.io.EOFException
Caused by: org.apache.catalina.connector.ClientAbortException: java.io.EOFException
Caused by: java.io.EOFException: null
2024-09-12T08:20:21.331Z [jfrt ] [ERROR] [35d5a484b17b6f0a] [c.e.m.GlobalExceptionMapper:51] [ttp-nio-8081-exec-97] - org.jfrog.storage.StorageException: Failure during saveToTempFileAndUpload
org.artifactory.storage.StorageException: org.jfrog.storage.StorageException: Failure during saveToTempFileAndUpload
        at org.artifactory.sh.service.ArtifactoryBinaryServiceImpl.addBinary(ArtifactoryBinaryServiceImpl.java:140)
        at org.artifactory.repo.service.RepositoryServiceImpl.saveResource(RepositoryServiceImpl.java:2649)
        at jdk.internal.reflect.GeneratedMethodAccessor359.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:234)
        at jdk.proxy2/jdk.proxy2.$Proxy74.saveResource(Unknown Source)
        at org.artifactory.engine.UploadServiceImpl.uploadItemWithContent(UploadServiceImpl.java:795)
        at org.artifactory.engine.UploadServiceImpl.uploadItemWithProvidedContent(UploadServiceImpl.java:766)
        at org.artifactory.engine.UploadServiceImpl.uploadItem(UploadServiceImpl.java:629)
        at org.artifactory.engine.UploadServiceImpl.uploadFile(UploadServiceImpl.java:618)
        at org.artifactory.engine.UploadServiceImpl.uploadArtifact(UploadServiceImpl.java:597)
        at org.artifactory.engine.UploadServiceImpl.adjustResponseAndUpload(UploadServiceImpl.java:335)
        at org.artifactory.engine.UploadServiceImpl.validateRequestAndUpload(UploadServiceImpl.java:290)
        at org.artifactory.engine.UploadServiceImpl.processUpload(UploadServiceImpl.java:228)
        at org.artifactory.api.request.GracefulShutdownAware.track(GracefulShutdownAware.java:36)
        at org.artifactory.engine.UploadServiceImpl.upload(UploadServiceImpl.java:210)
        at jdk.internal.reflect.GeneratedMethodAccessor351.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)

What you expected to happen: the image pushes successfully

How to reproduce it (as minimally and precisely as possible): docker push artifactory-url/image-name

Anything else we need to know: No support, so we pretty much have to fix this on our own

Sep 12 '24 08:09 RahmanBadru

Hi @RahmanBadru. Can you please enable debug logging for org.jfrog.type.s3? This way we may be able to have better visibility of what may have caused this. Thanks.

<logger name="org.jfrog.type.s3" level="debug"/>

Sep 12 '24 08:09 arikogan

Here is what i can see

Failed while saving temp file /opt/jfrog/artifactory/var/data/artifactory/filestore/_pre/dbRecord1770488452772219306-40f476e121806379-docker-local.bin
2024-09-12T10:54:35.729Z [jfrt ] [DEBUG] [40f476e121806379] [.j.t.s.S3AwsBinaryProvider:283] [http-nio-8081-exec-3] -
org.jfrog.storage.binstore.ifc.ClientInputStreamException: Failed to read stream: Failed to read stream: java.io.EOFException
        at org.jfrog.storage.binstore.ifc.ClientStream.read(ClientStream.java:36)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1487)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:1107)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1460)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:1085)
        at org.apache.commons.io.FileUtils.copyToFile(FileUtils.java:1026)
        at org.apache.commons.io.FileUtils.copyInputStreamToFile(FileUtils.java:934)
        at org.jfrog.storage.binstore.providers.tools.FilePersistenceHelper.saveStreamToTempFile(FilePersistenceHelper.java:52)
        at org.jfrog.type.s3.S3AwsBinaryProvider.saveToTempFileAndUpload(S3AwsBinaryProvider.java:269)
        at org.jfrog.type.s3.S3AwsBinaryProvider.addStream(S3AwsBinaryProvider.java:236)
        at org.jfrog.storage.binstore.providers.cachefs.FileCacheBinaryProviderImpl.addStream(FileCacheBinaryProviderImpl.java:225)
        at org.jfrog.storage.binstore.providers.federated.FederatedBinaryProvider.addStream(FederatedBinaryProvider.java:360)
        at org.jfrog.storage.binstore.BinaryServiceImpl.addStreamToBinaryProviderIfNeeded(BinaryServiceImpl.java:1342)
        at org.jfrog.storage.binstore.BinaryServiceImpl.addBinary(BinaryServiceImpl.java:435)
        at org.jfrog.storage.binstore.BinaryServiceImpl.addBinary(BinaryServiceImpl.java:422)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
--
        at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:312)
        at org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:615)
        at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:358)
        at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:132)
        at org.glassfish.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:79)
        at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:205)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
        at org.jfrog.storage.binstore.common.Sha1Sha2Md5ChecksumInputStream.read(Sha1Sha2Md5ChecksumInputStream.java:134)
        at org.jfrog.storage.binstore.common.Sha1Sha2Md5ChecksumInputStream.read(Sha1Sha2Md5ChecksumInputStream.java:118)
        at org.jfrog.storage.binstore.ifc.ClientStream.read(ClientStream.java:34)
        ... 145 common frames omitted
Caused by: java.io.EOFException: null
        at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1320)
        at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1231)
        at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:789)
        at org.apache.coyote.http11.Http11InputBuffer.access$400(Http11InputBuffer.java:41)
        at org.apache.coyote.http11.Http11InputBuffer$SocketInputBuffer.doRead(Http11InputBuffer.java:1195)
        at org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:254)
        at org.apache.coyote.http11.filters.ChunkedInputFilter.fill(ChunkedInputFilter.java:295)
        at org.apache.coyote.http11.filters.ChunkedInputFilter.parseChunkHeader(ChunkedInputFilter.java:328)
        at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:136)
        at org.apache.coyote.http11.Http11InputBuffer.doRead(Http11InputBuffer.java:244)
        at org.apache.coyote.Request.doRead(Request.java:626)
        at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:303)
        ... 156 common frames omitted

Sep 12 '24 10:09 RahmanBadru

Thank you for this. It's not related to the Helm Charts but seems to be something with your client interrupting the connection. Please contact our Support.

Sep 12 '24 11:09 arikogan

Hi, currently dont have access to support, so would appreciate help

Sep 21 '24 13:09 RahmanBadru

@RahmanBadru, I replied on your Stack Overflow question, let's correspond there.

Sep 22 '24 13:09 arikogan

Hi @ArielKogan, I have responded

On Sun, Sep 22, 2024 at 14:09 Ariel Kogan @.***> wrote:

@RahmanBadru https://github.com/RahmanBadru, I replied on your Stack Overflow question, let's correspond there.

— Reply to this email directly, view it on GitHub https://github.com/jfrog/charts/issues/1921#issuecomment-2366781587, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI5RREGKATY6TJSZWBHVUTTZX26P3AVCNFSM6AAAAABOCUELDSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRWG44DCNJYG4 . You are receiving this because you were mentioned.Message ID: @.***>

Sep 22 '24 13:09 RahmanBadru