DynamoDBMapperConfig.PaginationLoadingStrategy.ITERATION_ONLY is not compatible with scan queries

[sshevlyagin]

Expected Behavior

It's possible to run queries that filter by a value with DynamoDBMapperConfig.PaginationLoadingStrategy.ITERATION_ONLY

Actual Behavior

You get the following error:

java.lang.UnsupportedOperationException: The list could only be iterated once in ITERATION_ONLY mode.
	at com.amazonaws.services.dynamodbv2.datamodeling.PaginatedList$PaginatedListIterator.<init>(PaginatedList.java:231)
	at com.amazonaws.services.dynamodbv2.datamodeling.PaginatedList.iterator(PaginatedList.java:204)
	at java.util.Spliterators$IteratorSpliterator.estimateSize(Spliterators.java:1821)

Steps to Reproduce the Problem

1. Set up the following DynamoDBMapperConfig

    @Bean
    public DynamoDBMapperConfig dynamoDBMapperConfig() {
        return new DynamoDBMapperConfig.Builder()
       .withSaveBehavior(DynamoDBMapperConfig.SaveBehavior.UPDATE_SKIP_NULL_ATTRIBUTES)
       .withConsistentReads(DynamoDBMapperConfig.ConsistentReads.EVENTUAL)
.withPaginationLoadingStrategy(DynamoDBMapperConfig.PaginationLoadingStrategy.ITERATION_ONLY)
.withBatchWriteRetryStrategy(DynamoDBMapperConfig.DefaultBatchWriteRetryStrategy.INSTANCE)
.withBatchLoadRetryStrategy(DynamoDBMapperConfig.DefaultBatchLoadRetryStrategy.INSTANCE)
.withTypeConverterFactory(DynamoDBTypeConverterFactory.standard())
.withConversionSchema(ConversionSchemas.V2)
.build();

    }

1. In your repository have a findBy method

@EnableScan
public interface FooRepository extends CrudRepository<FooTableItem, String>{
    Iterable<FooItem> findByStatus(String status);
}

1. Execute the query and try to iterate over the results

private void doIt {
        Iterable<FooItem> items = fooRepository.findByStatus("NEW");
        List<Bar> results = new ArrayList<>();
        items.forEach(item -> {
            try {
                results.add(item.toContext());
            } catch (Throwable t) {
                log.error("Failed to deserialize failure", t);
            }
        });
        return results;
    }

Note: Setting a breakpoint on https://github.com/aws/aws-sdk-java/blob/1.11.794/aws-java-sdk-dynamodb/src/main/java/com/amazonaws/services/dynamodbv2/datamodeling/PaginatedList.java#L230 will show you that the PaginatedListIterator was called before the forLoop got there with the following stack.

<init>:234, PaginatedList$PaginatedListIterator (com.amazonaws.services.dynamodbv2.datamodeling)
iterator:204, PaginatedList (com.amazonaws.services.dynamodbv2.datamodeling)
requiresConversion:188, QueryExecutionResultHandler (org.springframework.data.repository.core.support)
postProcessInvocationResult:157, QueryExecutionResultHandler (org.springframework.data.repository.core.support)
postProcessInvocationResult:77, QueryExecutionResultHandler (org.springframework.data.repository.core.support)
invoke:605, RepositoryFactorySupport$QueryExecutorMethodInterceptor (org.springframework.data.repository.core.support)
proceed:186, ReflectiveMethodInvocation (org.springframework.aop.framework)
invoke:95, ExposeInvocationInterceptor (org.springframework.aop.interceptor)
proceed:186, ReflectiveMethodInvocation (org.springframework.aop.framework)
invoke:212, JdkDynamicAopProxy (org.springframework.aop.framework)
findByStatus:-1, $Proxy154 (com.sun.proxy)
-REDACTED-
-REDACTED-
invoke0:-1, NativeMethodAccessorImpl (sun.reflect)
invoke:62, NativeMethodAccessorImpl (sun.reflect)
invoke:43, DelegatingMethodAccessorImpl (sun.reflect)
invoke:498, Method (java.lang.reflect)
run:84, ScheduledMethodRunnable (org.springframework.scheduling.support)
run:54, DelegatingErrorHandlingRunnable (org.springframework.scheduling.support)
call:511, Executors$RunnableAdapter (java.util.concurrent)
runAndReset$$$capture:308, FutureTask (java.util.concurrent)
runAndReset:-1, FutureTask (java.util.concurrent)
 - Async stack trace
<init>:151, FutureTask (java.util.concurrent)
<init>:219, ScheduledThreadPoolExecutor$ScheduledFutureTask (java.util.concurrent)
scheduleAtFixedRate:570, ScheduledThreadPoolExecutor (java.util.concurrent)
scheduleAtFixedRate:348, ThreadPoolTaskScheduler (org.springframework.scheduling.concurrent)
scheduleFixedRateTask:479, ScheduledTaskRegistrar (org.springframework.scheduling.config)
scheduleFixedRateTask:453, ScheduledTaskRegistrar (org.springframework.scheduling.config)
scheduleTasks:374, ScheduledTaskRegistrar (org.springframework.scheduling.config)
afterPropertiesSet:349, ScheduledTaskRegistrar (org.springframework.scheduling.config)
finishRegistration:302, ScheduledAnnotationBeanPostProcessor (org.springframework.scheduling.annotation)
onApplicationEvent:233, ScheduledAnnotationBeanPostProcessor (org.springframework.scheduling.annotation)
onApplicationEvent:105, ScheduledAnnotationBeanPostProcessor (org.springframework.scheduling.annotation)
doInvokeListener:172, SimpleApplicationEventMulticaster (org.springframework.context.event)
invokeListener:165, SimpleApplicationEventMulticaster (org.springframework.context.event)
multicastEvent:139, SimpleApplicationEventMulticaster (org.springframework.context.event)
publishEvent:403, AbstractApplicationContext (org.springframework.context.support)
publishEvent:360, AbstractApplicationContext (org.springframework.context.support)
finishRefresh:897, AbstractApplicationContext (org.springframework.context.support)
finishRefresh:162, ServletWebServerApplicationContext (org.springframework.boot.web.servlet.context)
refresh:553, AbstractApplicationContext (org.springframework.context.support)
refresh:141, ServletWebServerApplicationContext (org.springframework.boot.web.servlet.context)
refresh:747, SpringApplication (org.springframework.boot)
refreshContext:397, SpringApplication (org.springframework.boot)
run:315, SpringApplication (org.springframework.boot)
run:1226, SpringApplication (org.springframework.boot)
run:1215, SpringApplication (org.springframework.boot)
REDACTED

Specifications

• Spring Data DynamoDB Version: 5.2.4 (2.2)
• Spring Data Version: 2.2.7.RELEASE
• AWS SDK Version: 1.11.790
• Java Version: 1.8.0_181 - Java HotSpot(TM) 64-Bit Server VM 25.181-b13
• Platform Details: Mac OS X 10.15.5

[boostchicken]

I have been looking into this for a while. So far what I have found is Spring Data made some changes upstream regarding their conversion service. This means the query get's iterated before it gets back to you. I have not been successful in finding a way to get it to not think that these results are candidates for conversion.

I'll leave this issue open for tracking purposes.

[boostchicken]

@sshevlyagin https://github.com/spring-projects/spring-data-commons/blob/210ab249e373c535796b1ff6f9ccfe1ab9c55a7b/src/main/java/org/springframework/data/repository/core/support/QueryExecutionResultHandler.java#L175-L196

You can see there that they loop through the collection before the result ever gets back to us to verify it does not have to do any type conversion.

[sshevlyagin]

Thanks! Is that a bug on the part of SpringData or is ITERATION_ONLY simply not compatible anymore?

[boostchicken]

for the moment I am going to say ITERATION_ONLY is simply not compatible anymore. Spring Data has changed a lot and if i circumvent this stuff its going to break a lot of expected functionality