dynamodb-toolbox
dynamodb-toolbox copied to clipboard
jeremydaly
Query and Scan do not parse `sets`.
We have noticed that while using the functions scan and query any set attributes do not seem to be parsed correctly. They come back as a DynamoDBSet and not as a list. However, when we use the get function, all the attributes that are returned appear to have been parsed correctly (including sets).
Is this by design? Perhaps I missed a setting. Any advice would be greatly appreciated.
Get result:
{ id: 'id', someSet: [ 'one', 'two' ] }
Scan and Query result:
{ id: 'id', someSet: [Set] }
I've played around with some data and found the following:
-
when querying/scanning a table, each item in the table needs the '_et' (entity name) field in order for it to be parsed properly, because we're checking the mapping of that specific entity.
-
when using the
getoperation, because the scope is a specific entity, we're looking at that entity's mapping and parse the fields according to it, even when it doesn't have the '_et' field explicitly.
So I'm guessing that what happened was that somehow the '_et' field was missing.
What we can do is if someone attempts to query a table using the entity (Entity.query(...)), we'll enforce the mapping check to be against that specific entity's mapping, imo it's not something that I'd expect the library to do.
Another solution would be to ignore the mapping and convert the property's value to an array if its .values is an array, which is basically checking wether the value is a DDBSet.
What do you think? @jeremydaly @krackerman
@naorpeled Thank you for taking a look.
Just so I understand what is happening. get will automatically parse because it is scoped to that specific Entity. Like a Customer as an example.
If you are using a Single Table desgin, Scan and Query can return different entities (Customers, Businesses, Orders, etc). So Scan and Query will not automatically parse if the _et field is not set on that Entity. Is this true?
We are not using a Single Table design so we did not set the _et field. So this feels like we are just missing setting.
@naorpeled Thank you for taking a look.
Just so I understand what is happening.
getwill automatically parse because it is scoped to that specificEntity. Like a Customer as an example. If you are using a Single Table desgin,ScanandQuerycan return different entities (Customers, Businesses, Orders, etc). So Scan and Query will not automatically parse if the_etfield is not set on that Entity. Is this true?We are not using a Single Table design so we did not set the
_etfield. So this feels like we are just missing setting.
Exactly.
@naorpeled This is exactly what I needed thank you for all the help.
You're welcome! :)
I'll be closing this issue for now.
Looks like this is not working as expected. We are trying to add entity into scan and query, but it doesn't seem to fix the issue.
entity: The name of a table Entity to evaluate filters and attributes against.
Am I missing something here? Will look into creating a minimal test case next.
For reference, this doesn't work: https://github.com/blackflux/aws-sdk-wrap/pull/2151/files
We'll try and get a minimal test case in so it's easier to reason about. Don't have an eta for that though
Looks like this is not working as expected. We are trying to add entity into
scanandquery, but it doesn't seem to fix the issue.
entity: The name of a table Entity to evaluate filters and attributes against.Am I missing something here? Will look into creating a minimal test case next.
I see that in the code, we're using the entity name for building filter expressions and projections but not for parsing the attributes of the returned items.
Not sure whether I'd change that logic because you might have several entities in your table and enforcing the parsing of all items would potentially break the parsing of the entities that are not provided in the entity parameter.
At the moment what you can do is add the entity name (_et by default) field with the entity name and it would be parsed properly, just make sure to map the specific set field's type to set.
Wdyt?
At the moment what you can do is add the entity name (_et by default) field with the entity name and it would be parsed properly, just make sure to map the specific set field's type to
set.
@naorpeled Just so I understand - we would have to rewrite all our millions of dynamodb records to accomplish this, correct?
I would be great if there was a way to tell dynamodb-toolbox to "parse all responses as a certain entity". Maybe this is a separate option that is passed in?
Edit: Maybe a parseAs option?
At the moment what you can do is add the entity name (_et by default) field with the entity name and it would be parsed properly, just make sure to map the specific set field's type to
set.@naorpeled Just so I understand - we would have to rewrite all our millions of dynamodb records to accomplish this, correct?
I would be great if there was a way to tell dynamodb-toolbox to "parse all responses as a certain entity". Maybe this is a separate option that is passed in?
Edit: Maybe a
parseAsoption?
Unfortunately with the current implementation that is correct.
Will discuss this with @jeremydaly and see what fits best for this scenario.
Sorry for being late to this conversation, but any Entity written to the table using the toolbox should get an _et value (unless you disabled it). If it doesn't have that value, then, yes, we would need to add some sort of parseAs option, but the idea is not to need that. Let me dig into the code a bit and see if we're missing something else here.