alluxio icon indicating copy to clipboard operation
alluxio copied to clipboard

aws-java-sdk-s3 APIS. get object from alluxio, filecheck is failed

Open wsjswy opened this issue 2 years ago • 1 comments

Alluxio Version: What version of Alluxio are you using? v2.8.0

Describe the bug aws-java-sdk-s3 APIS. get data from alluxio, filecheck is failed

for example S3-client-demo: init s3

AmazonS3 s3Client = AmazonS3ClientBuilder .standard() .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://127.0.0.1:39999/api/v1/s3/", Regions.US_EAST_1.name())) .withPathStyleAccessEnabled(true) .withClientConfiguration(clientConfiguration) .withCredentials(new AWSStaticCredentialsProvider(credentials)) .build(); S3-client-sdk: `if (!this.skipMd5CheckStrategy.skipClientSideValidation(getObjectRequest, s3Object.getObjectMetadata())) { byte[] serverSideHash = BinaryUtils.fromHex(s3Object.getObjectMetadata().getETag());

try { MessageDigest digest = MessageDigest.getInstance("MD5"); is = new DigestValidationInputStream((InputStream)is, digest, serverSideHash); } catch (NoSuchAlgorithmException var11) { log.warn("No MD5 digest algorithm available. Unable to calculate checksum and verify data integrity.", var11); } } else { is = new LengthCheckInputStream(progressInputStream, s3Object.getObjectMetadata().getContentLength(), true); }`

s3Object.getObjectMetadata().getETag() should be a hex hash value

For alluxio-proxy-s3-service:

Response.ResponseBuilder res = Response.ok(ris) .lastModified(new Date(status.getLastModificationTimeMs())) .header(S3Constants.S3_ETAG_HEADER, "\"" + status.getLastModificationTimeMs() + "\"") .header(S3Constants.S3_CONTENT_LENGTH_HEADER, s3Range.getLength(status.getLength()));

use status.getLastModificationTimeMs() instead

wsjswy avatar Jul 20 '22 06:07 wsjswy

@ZhuTopher can you take a look? @jingwenouyang FYI

HelloHorizon avatar Jul 21 '22 16:07 HelloHorizon

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Feb 01 '23 15:02 github-actions[bot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Mar 20 '23 15:03 github-actions[bot]