tempest
tempest copied to clipboard
Pagination doesn't work when querying a GSI with prefixed sort key
We encountered a failing non-null assertion in tempest's Prefixer
when performing a query on a GSI using a prefixed sort key, once the result set grew large enough to require pagination, causing our service to be unable to read from the database.
Analysis by @jialiang-cash found that the LastEvaluatedKey
object returned from Dynamo only includes the partition_key
and secondary_index_keys
fields. This caused an issue in decodeOffset
when Tempest tries to remove the prefixes on both partition_key
and sort_key
, resulting in a null value exception because no sort_key
exists.
Stack trace:
java.lang.IllegalArgumentException: Required value was null.
at app.cash.tempest.internal.ReflectionCodec$Prefixer.removePrefix(Codec.kt:137)
at app.cash.tempest.internal.ReflectionCodec.toApp(Codec.kt:92)
at app.cash.tempest2.internal.DynamoDbQueryable.decodeOffset(DynamoDbQueryable.kt:153)
at app.cash.tempest2.internal.DynamoDbQueryable.toQueryResponse(DynamoDbQueryable.kt:114)
at app.cash.tempest2.internal.DynamoDbQueryable.access$toQueryResponse(DynamoDbQueryable.kt:39)
at app.cash.tempest2.internal.DynamoDbQueryable$Sync.query(DynamoDbQueryable.kt:68)
at app.cash.tempest2.internal.LogicalDbFactory$SecondaryIndexFactory$secondaryIndex$1.query(LogicalDbFactory.kt)
at app.cash.tempest2.Queryable$DefaultImpls.query$default(Query.kt:27)
This is the assertion that's failing: https://github.com/cashapp/tempest/blob/3f520efd7a9f1c5b341ea51a7efcb403e609cee0/tempest-internal/src/main/kotlin/app/cash/tempest/internal/Codec.kt#L137