tempest icon indicating copy to clipboard operation
tempest copied to clipboard

Pagination doesn't work when querying a GSI with prefixed sort key

Open salamagd opened this issue 2 years ago • 0 comments

We encountered a failing non-null assertion in tempest's Prefixer when performing a query on a GSI using a prefixed sort key, once the result set grew large enough to require pagination, causing our service to be unable to read from the database.

Analysis by @jialiang-cash found that the LastEvaluatedKey object returned from Dynamo only includes the partition_key and secondary_index_keys fields. This caused an issue in decodeOffset when Tempest tries to remove the prefixes on both partition_key and sort_key, resulting in a null value exception because no sort_key exists.

Stack trace:

java.lang.IllegalArgumentException: Required value was null.
	at app.cash.tempest.internal.ReflectionCodec$Prefixer.removePrefix(Codec.kt:137)
	at app.cash.tempest.internal.ReflectionCodec.toApp(Codec.kt:92)
	at app.cash.tempest2.internal.DynamoDbQueryable.decodeOffset(DynamoDbQueryable.kt:153)
	at app.cash.tempest2.internal.DynamoDbQueryable.toQueryResponse(DynamoDbQueryable.kt:114)
	at app.cash.tempest2.internal.DynamoDbQueryable.access$toQueryResponse(DynamoDbQueryable.kt:39)
	at app.cash.tempest2.internal.DynamoDbQueryable$Sync.query(DynamoDbQueryable.kt:68)
	at app.cash.tempest2.internal.LogicalDbFactory$SecondaryIndexFactory$secondaryIndex$1.query(LogicalDbFactory.kt)
	at app.cash.tempest2.Queryable$DefaultImpls.query$default(Query.kt:27)

This is the assertion that's failing: https://github.com/cashapp/tempest/blob/3f520efd7a9f1c5b341ea51a7efcb403e609cee0/tempest-internal/src/main/kotlin/app/cash/tempest/internal/Codec.kt#L137

salamagd avatar Oct 20 '22 15:10 salamagd