pplu
DynamoDB.pm - issue with keys
Hello,
I'm trying to write some code to get data from a table in Dynamo. As you know AWS limits the records returned so you sometimes need to do an exponential back-off to get the rest of the records. Also, in some cases you may want to use Limit to help with pagination. In any case, I am having issues using ExclusiveStartKey to retrieve subsequent pages of data.
sub getData {
my $filter = shift;
my @records = ();
my $key = undef;
my $options = {
TableName => 'my-table',
ProjectionExpression => 'id, field1, field2',
FilterExpression => 'field1 = :filterPlaceholder',
ExpressionAttributeValues => {
':filterPlaceholder' => { S => $filter }
},
Limit => 10
};
do {
$options->{ExclusiveStartKey} = $key if $key;
my $result = $dynamo->Scan(%$options);
if ($result->Count > 0) {
push(@records, @{$result->Items});
$key = $result->LastEvaluatedKey;
}
} while (defined($key));
}
I am getting the following error after the first page of results is retrieved:
Paws::DynamoDB::AttributeValue doesn't have an id at /usr/share/perl5/Paws/DynamoDB.pm line 100.
I do not see any examples of this type of call within the documentation, so I'm a little unclear about what is going on and whether or not it is a bug or a misconfiguration on my part. The same code ported to nodejs works fine with the properties set above.
Hi,
Try to dump $options before and after the call to Scan. I'm afraid that Scan may be modifying things in the options hashref, causing an "id" key to appear, and thus breaking the second call. I think that deep cloning $options before calling Scan should solve your problem.
For anyone else who searched about error messages when using "ExclusiveStartKey" and wound up here, there are two reported issues that are important:
- https://github.com/pplu/aws-sdk-perl/issues/286
- https://github.com/pplu/aws-sdk-perl/issues/79
Issue 286 indicates that you cannot simply use $result->LastEvaluatedKey as the argument to ExclusiveStartKey on the next loop for pagination. Instead, you must use $result->LastEvaluatedKey->Map. HOWEVER, Issue 79 indicates that there is an empty 'L' attribute which will cause an error, namely:
The provided starting key is invalid: Supplied AttributeValue has more than one datatypes set,
must contain exactly one of the supported datatypes
This can be fixed by iterating over the AttributeMap and deleting the extra empty 'L' elements. For example:
$lastkey = $result->LastEvaluatedKey->Map;
foreach my $key (keys %{$lastkey}) {
delete($lastkey->{$key}->{L});
}
On the next call to Scan() or Query(), you can now set ExclusiveStartKey => $lastkey.
Note that ScanAll() and QueryAll() do not perform this extra processing of the LastEvaluatedKey (at the time of writing this comment), so you either have to use Scan() or Query() and process pagination on your own, or patch /usr/share/perl5/Paws/DynamoDB.pm to massage LastEvaluatedKey as above. While you are patching, be sure to apply the patch from https://github.com/pplu/aws-sdk-perl/issues/333 , otherwise ScanAll() and QueryAll() don't work at all.
Attached is my (admittedly naive) attempt at patching DynamoDB.pm , which also incorporates the patch from https://github.com/pplu/aws-sdk-perl/issues/333 . I hope someone with better Perl-fu can do a nicer job (and hopefully incorporate it into the next release).
