aws-sdk-java icon indicating copy to clipboard operation
aws-sdk-java copied to clipboard
verified publisher icon aws

Getting AEADBadTagException while fetching previously uploaded object from S3

Open trznjak opened this issue 4 years ago • 3 comments

Describe the bug

We are using AmazonS3EncryptionClient to persist objects to the S3 bucket and since we switched to the Bouncy Castle (BC) FIPS provider, we started seeing this issue in our Production environment. We are using the S3 Client in AuthenticatedEncryption Crypto Mode (AES/GCM/NoPadding). I have created a small test app that mimics our server behavior and managed to reproduce the issue. When I switch to the regular (non-FIPS) BC version of the provider, I am not able to reproduce the issue. As a workaround, I am putting locks around the S3 clients putObject() method but it has a huge performance impact.

Expected Behavior

I am expecting that the S3 Client is thread-safe.

Current Behavior

I am intermittently getting the AEADBadTagException when I try to fetch previously uploaded objects from the S3 bucket.

Exception: Exception in thread "Thread-2617" java.lang.SecurityException: javax.crypto.AEADBadTagException: Error finalising cipher data: mac check in GCM failed at com.amazonaws.services.s3.internal.crypto.CipherLiteInputStream.nextChunk(CipherLiteInputStream.java:238) at com.amazonaws.services.s3.internal.crypto.CipherLiteInputStream.readNextChunk(CipherLiteInputStream.java:118) at com.amazonaws.services.s3.internal.crypto.CipherLiteInputStream.read(CipherLiteInputStream.java:95) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90) at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185) at java.base/java.io.BufferedReader.fill(BufferedReader.java:161) at java.base/java.io.BufferedReader.readLine(BufferedReader.java:326) at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392) at com.adobe.DIVtest.DivTestApplication.getObject(DivTestApplication.java:136) at com.adobe.DIVtest.DivTestApplication.lambda$uploadAndDownloadS3ObjectsInThreads$0(DivTestApplication.java:164) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: javax.crypto.AEADBadTagException: Error finalising cipher data: mac check in GCM failed at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at org.bouncycastle.jcajce.provider.ClassUtil.throwBadTagException(Unknown Source) at org.bouncycastle.jcajce.provider.BaseCipher.engineDoFinal(Unknown Source) at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2085) at com.amazonaws.services.s3.internal.crypto.CipherLite.doFinal(CipherLite.java:172) at com.amazonaws.services.s3.internal.crypto.GCMCipherLite.doFinal(GCMCipherLite.java:99) at com.amazonaws.services.s3.internal.crypto.CipherLiteInputStream.nextChunk(CipherLiteInputStream.java:226) ... 13 more

Steps to Reproduce

Create a simple spring boot application (my version: 2.4.2). Add dependencies to the pom.xml:

		<dependency>
			<groupId>com.amazonaws</groupId>
			<artifactId>aws-java-sdk-s3</artifactId>
			<version>1.11.970</version>
		</dependency>

		<dependency>
			<groupId>org.bouncycastle</groupId>
			<artifactId>bc-fips</artifactId>
			<version>1.0.2</version>
		</dependency>

Here is the code that reproduces the issue:

package com.adobe.DIVtest;

import com.amazonaws.AmazonClientException;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.SdkClientException;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3EncryptionClientBuilder;
import com.amazonaws.services.s3.model.*;
import org.apache.http.HttpStatus;
import org.bouncycastle.crypto.CryptoServicesRegistrar;
import org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider;
import org.springframework.boot.autoconfigure.SpringBootApplication;

import javax.crypto.*;
import javax.crypto.spec.SecretKeySpec;
import java.io.*;
import java.security.*;
import java.time.LocalDateTime;
import java.time.ZoneOffset;
import java.util.ArrayList;
import java.util.List;

import static com.amazonaws.services.s3.model.CryptoMode.AuthenticatedEncryption;

@SpringBootApplication
public class DivTestApplication {

	private static final String BUCKET_NAME = "YOUR_BUCKET_NAME"; // TODO: insert your bucket name
	// for ACCESS_KEY and SECRET_KEY you need to fallow instructions on below page, look at "Create an IAM user" section
	// https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html#get-started-setup-user
	private static final String ACCESS_KEY = "YOUR_ACCESS_KEY"; // TODO: insert your access key
	private static final String SECRET_KEY = "YOUR_SECRET_KEY"; // TODO: insert your secret key
	private static final Integer THREAD_COUNT = 3500;

	private static final Provider provider;
	private static AmazonS3 s3encryptionClient;

	static {
		provider = new BouncyCastleFipsProvider();
	}


	public static void main(String[] args) throws NoSuchAlgorithmException {
		LocalDateTime startDatetime;
		LocalDateTime endDateTime;

		startDatetime = LocalDateTime.now();
		System.out.println("Start time: " + startDatetime);
		System.out.println("Is in Approved mode: " + CryptoServicesRegistrar.isInApprovedOnlyMode());


		uploadAndDownloadS3ObjectsInThreads();
//		deleteS3ObjectsInThreads();  // uncomment this when you want to clean the test data from S3 bucket


		endDateTime = LocalDateTime.now();
		System.out.println("End Time: " + endDateTime);
		System.out.println("Duration: " + (endDateTime.toEpochSecond(ZoneOffset.UTC) - startDatetime.toEpochSecond(ZoneOffset.UTC) + " seconds." ));
	}

	public static AmazonS3 getS3Client() throws NoSuchAlgorithmException, UnsupportedEncodingException {

		if (s3encryptionClient == null) {
			System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "1");
			System.setProperty("com.amazonaws.services.s3.disablePutObjectMD5Validation", "1");

			SecureRandom fipsSecureRandom = SecureRandom.getInstance("NONCEANDIV", provider); // NONCEANDIV

			CryptoConfiguration cryptoConfig = new CryptoConfiguration();
			cryptoConfig.setSecureRandom(fipsSecureRandom);
			cryptoConfig.setAlwaysUseCryptoProvider(true);
			cryptoConfig.setCryptoProvider(provider);
			cryptoConfig.setCryptoMode(AuthenticatedEncryption);

			//Generate secret key.
			SecretKey key = new SecretKeySpec("19Mv6GiAUVhKz3nMPReIMrfz2bDSyol0".getBytes("UTF-8"), "AES");
			EncryptionMaterialsProvider emProvider = new StaticEncryptionMaterialsProvider(new EncryptionMaterials(key));


			s3encryptionClient = AmazonS3EncryptionClientBuilder.standard()
					.withEncryptionMaterials(emProvider)
					.withCryptoConfiguration(cryptoConfig)
					.withRegion(Regions.US_EAST_1)
					.withClientConfiguration(new ClientConfiguration().withMaxConnections(3500))
					.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY)))
					.build();
		}
		return s3encryptionClient;
	}

	public static void putObject(String s3ObjectPath) throws IOException, NoSuchAlgorithmException {
		AmazonS3 client = getS3Client();

		try {
			client.putObject(BUCKET_NAME, s3ObjectPath, "test");
		} catch (AmazonServiceException ase) {
			// Caught an AmazonServiceException, which means the request made it
			// to Amazon S3, but was rejected with an error response for some reason
			System.out.println("Failed to save data to S3 due to error response. " + ase);
			throw ase;
		} catch (AmazonClientException ace) {
			// Caught an AmazonClientException, which means the client encountered
			// an internal error while trying to communicate with S3, such as not
			// being able to access the network.
			System.out.println("Failed to save data to S3 due to client error. " + ace);
			throw ace;
		}
	}

	public static String getObject(String s3ObjectPath) throws IOException, NoSuchAlgorithmException {
		AmazonS3 client = getS3Client();
		S3Object s3object = null;
		try {
			s3object = client.getObject(new GetObjectRequest(BUCKET_NAME, s3ObjectPath));
		} catch (AmazonS3Exception e) {
			if (e.getStatusCode() != HttpStatus.SC_NOT_FOUND &&
					e.getStatusCode() != HttpStatus.SC_FORBIDDEN) {
				System.out.println("Failed to retrieve data from S3. " + e);
				throw e;
			}
		}

		if (s3object == null) {
			return "OBJECT NOT FOUND";
		}

		try (InputStream stream = s3object.getObjectContent(); ) {
			// Read the whole content of the stream as a string
			BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
			StringBuilder sb = new StringBuilder();
			while (true) {
				String line = reader.readLine();
				if (line == null) {
					break;
				}
				sb.append(line);
				sb.append('\n');
			}
			s3object.close();
			return sb.toString();
		} catch (IOException ioe) {
			// got problems when reading from the content stream
			System.out.println("Failed to retrieve data from S3 object content stream. " + ioe);
			throw ioe;
		}
	}


	public static void uploadAndDownloadS3ObjectsInThreads() throws NoSuchAlgorithmException {
		List<Thread> threadList = new ArrayList<>();
		for(int x = 0; x < THREAD_COUNT; x++) {
			Thread tmp = new Thread(() -> {
				try {
					System.out.println(Thread.currentThread().getName() + " - Put Object on this path: " + Thread.currentThread().getName());
					putObject(Thread.currentThread().getName());
					System.out.println(Thread.currentThread().getName() + " - Object put on this path: " + Thread.currentThread().getName());


					System.out.println(Thread.currentThread().getName() + " - Getting Object from this path: " + Thread.currentThread().getName());
					String response = getObject(Thread.currentThread().getName());
					System.out.println(Thread.currentThread().getName() + " - Object retrieved from this path: " + Thread.currentThread().getName() + " Response String: " + response );
				} catch ( NoSuchAlgorithmException e) {
					e.printStackTrace();
				} catch (IOException | SdkClientException e){
					System.out.println("Thread: " + Thread.currentThread().getName() + " Exception: " + e.toString());
				}
			});
			tmp.start();
			threadList.add(tmp);
		}

		for(Thread t : threadList){
			try {
				t.join();
			} catch (InterruptedException e) {
				e.printStackTrace();
			}
		}
	}

	public static void deleteObject (String s3ObjectPath) throws NoSuchAlgorithmException, UnsupportedEncodingException {
		AmazonS3 client = getS3Client();
		DeleteObjectRequest delRequest = new DeleteObjectRequest(BUCKET_NAME, s3ObjectPath);
		client.deleteObject(delRequest);
	}

	public static void deleteS3ObjectsInThreads() throws NoSuchAlgorithmException, InterruptedException {
		deleteRangeOfObjects(0, 1000);
		Thread.sleep(60000);
		deleteRangeOfObjects(1000,2000);
		Thread.sleep(60000);
		deleteRangeOfObjects(2000,3000);
		Thread.sleep(60000);
		deleteRangeOfObjects(3000,3500);
	}

	private static void deleteRangeOfObjects(int start, int end) {
		List<Thread> threadList = new ArrayList<>();
		for (int i = start; i < end; i++) {
			Thread tmp = new Thread(() -> {
				try {
					deleteObject(Thread.currentThread().getName());
				} catch (NoSuchAlgorithmException e) {
					e.printStackTrace();
				} catch (SdkClientException | UnsupportedEncodingException e) {
					System.out.println("Thread: " + Thread.currentThread().getName() + " Exception: " + e.toString());
				}
			});
			tmp.start();
			threadList.add(tmp);
		}

		for(Thread t : threadList){
			try {
				t.join();
			} catch (InterruptedException e) {
				e.printStackTrace();
			}
		}
	}
}

Create a src/main/resources/logback.xml with the below content to reduce the logging level:

<configuration>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <!-- encoders are assigned the type
             ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <root level="warn">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

Possible Solution

Context

Your Environment

  • AWS Java SDK version used: aws-java-sdk-s3 1.11.970
  • JDK version used: 11.0.8 Zulu JDK
  • Operating System and version: Mac OS Mojave 10.14.6

trznjak avatar Mar 30 '21 13:03 trznjak

Hi @trznjak apologies for the delayed response. I'll investigate.

debora-ito avatar Apr 08 '21 23:04 debora-ito

@trznjak the error may be related to BC FIPS version and AES compliance. I'm still researching.

In the meantime, can you run the same test case using AmazonS3EncryptionClientV2 to see if you get the same errors? AmazonS3EncryptionClient was deprecated in favor of AmazonS3EncryptionClientV2.

Also, how big are the files you are uploading? Since you are calling putObject and getObject one after the other, I'm wondering if using the objectExists waiter before calling getObject would make a difference in this case.

debora-ito avatar Apr 12 '21 21:04 debora-ito

Hello @debora-ito, sorry for the late response.

I am not able to use the AmazonS3EncryptionClientV2 because of the similar issue we are seeing in this ticket. I am getting the below exception when trying to initialize the V2 Client with BC FIPS version. (using the workaround from the mentioned ticket).

Exception in thread "main" java.lang.UnsupportedOperationException: The Bouncy castle library jar is required on the classpath to enable authenticated encryption
	at com.amazonaws.services.s3.model.CryptoConfigurationV2.checkBountyCastle(CryptoConfigurationV2.java:374)
	at com.amazonaws.services.s3.model.CryptoConfigurationV2.checkCryptoMode(CryptoConfigurationV2.java:366)
	at com.amazonaws.services.s3.model.CryptoConfigurationV2.<init>(CryptoConfigurationV2.java:68)
	at com.amazonaws.services.s3.model.CryptoConfigurationV2.<init>(CryptoConfigurationV2.java:47)
	at com.amazonaws.services.s3.model.CryptoConfigurationV2.clone(CryptoConfigurationV2.java:446)
	at com.amazonaws.services.s3.AmazonS3EncryptionClientV2.validateConfigAndCreateReadOnlyCopy(AmazonS3EncryptionClientV2.java:160)
	at com.amazonaws.services.s3.AmazonS3EncryptionClientV2.<init>(AmazonS3EncryptionClientV2.java:110)
	at com.amazonaws.services.s3.AmazonS3EncryptionClientV2Builder.build(AmazonS3EncryptionClientV2Builder.java:101)
	at com.amazonaws.services.s3.AmazonS3EncryptionClientV2Builder.build(AmazonS3EncryptionClientV2Builder.java:23)
	at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
	at com.test.Utils.getS3ClientV2(Utils.java:269)
	at com.test.TestApplication.main(TestApplication.java:100)

I am able to reproduce it using a small strings: client.putObject(BUCKET_NAME, s3ObjectPath, "test");

trznjak avatar Apr 23 '21 15:04 trznjak