alluxio
alluxio copied to clipboard
aws-java-sdk-s3 APIS. get object from alluxio, filecheck is failed
Alluxio Version: What version of Alluxio are you using? v2.8.0
Describe the bug aws-java-sdk-s3 APIS. get data from alluxio, filecheck is failed
for example S3-client-demo: init s3
AmazonS3 s3Client = AmazonS3ClientBuilder .standard() .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://127.0.0.1:39999/api/v1/s3/", Regions.US_EAST_1.name())) .withPathStyleAccessEnabled(true) .withClientConfiguration(clientConfiguration) .withCredentials(new AWSStaticCredentialsProvider(credentials)) .build();
S3-client-sdk:
`if (!this.skipMd5CheckStrategy.skipClientSideValidation(getObjectRequest, s3Object.getObjectMetadata())) {
byte[] serverSideHash = BinaryUtils.fromHex(s3Object.getObjectMetadata().getETag());
try { MessageDigest digest = MessageDigest.getInstance("MD5"); is = new DigestValidationInputStream((InputStream)is, digest, serverSideHash); } catch (NoSuchAlgorithmException var11) { log.warn("No MD5 digest algorithm available. Unable to calculate checksum and verify data integrity.", var11); } } else { is = new LengthCheckInputStream(progressInputStream, s3Object.getObjectMetadata().getContentLength(), true); }`
s3Object.getObjectMetadata().getETag() should be a hex hash value
For alluxio-proxy-s3-service:
Response.ResponseBuilder res = Response.ok(ris) .lastModified(new Date(status.getLastModificationTimeMs())) .header(S3Constants.S3_ETAG_HEADER, "\"" + status.getLastModificationTimeMs() + "\"") .header(S3Constants.S3_CONTENT_LENGTH_HEADER, s3Range.getLength(status.getLength()));
use status.getLastModificationTimeMs() instead
@ZhuTopher can you take a look? @jingwenouyang FYI
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.