Working with nested attributes

[lwillmeth]

Are there docs or examples around defining and querying nested attributes? I see docs for updating nested attributes, but they don't include the entity definition. I also checked the put unit tests for an example.

I tried defining the entity like this:

const Foo = new Entity({
    name: 'Foo',
    attributes: {
        pk: { partitionKey: true },
        sk: { sortKey: true },
        data: { type: 'map' } // I'd like to fully define the shape of this attribute.
  },
  table: toolboxTable
});

I can write data using await Foo.put({ pk, sk, data: { findMe: true } }), and a dynamo scan shows the record exists as:

{
    "sk": "abc",
    "pk": "test-a",
    "_et": "Foo",
    "data": {
        "findMe": true
    }
}

But a query using the correct pk/sk and filters: [{ attr: 'data.findMe', eq: true }] fails to find the record. (commenting out the filter returns the record.)

I'm guessing that it's trying to match on a key called 'data.findMe' instead of a nested key at data.findMe, but I'm not sure how to fix that?

1. How should I define and query nested attributes?
2. Is it possible to use query filters on undefined attributes?

[lwillmeth]

I think I see the problem, at least with the way that I'm trying to filter on that nested attribute. dynamodb-toolbox generates a dynamo query using the full path, like:

ExpressionAttributeNames: { ..., '#attr1': 'data.findMe' },
ExpressionAttributeValues: { ..., ':attr1': true },
FilterExpression: '#attr1 = :attr1',

but dynamo needs each step of the path to be broken down, more like this:

ExpressionAttributeNames: { ..., '#attr1': 'data', '#attr2': 'findMe' },
ExpressionAttributeValues: { ..., ':attr2': true },
FilterExpression: '#attr1.#attr2 = :attr2',

Am I just using this wrong? Is there another way to specify the path in the toolbox query args?

[jeremydaly]

Hmm. This could be a bug, but I recall nested paths working just fine in the past. Let me look at this.

[lwillmeth]

Thanks! I'm totally open to user error here.

Is it possible to define the shape of a map's attributes? In my playground example I'm doing this:

export const Progress = new Entity({
    name: 'ProgressRecord',
    timestamps: false,
    attributes: {
        pk: { partitionKey: true, default: ({ attrs: { orgId, userId, updatedAt } }) => {
            const YYYY = updatedAt.substring(0, 4); // YYYY
            return orgId ? `${orgId}#${YYYY}` : `${userId}#${YYYY}`;
        }},
        sk: { sortKey: true, default: ({ attrs: { updatedAt, type, id, userId } }) => {
            const MMDD = updatedAt.substring(5, 10); // MM-DD
            return `${MMDD}#${type}#${id}#${userId}`;
        }},
        pk2: { partitionKey: 'byPk2', default: ({ attrs: { userId } }) => userId },
        sk2: { sortKey: 'byPk2', default: ({ attrs: { updatedAt, type, id } }) => `${updatedAt}#${type}#${id}` },
        attrs: { type: 'map', default: (attrs) => attrs }
  },
  table: reportsTable
});

Which creates records like this:

{
    "pk": "org-A#2020",
    "sk": "01-05#skill#skill-M#user-H",
    "pk2": "user-H",
    "sk2": "2020-01-05#skill#skill-M",
    "attrs": {
        "managed": true,
        "id": "skill-M",
        "state": 0,
        "type": "skill",
        "userId": "user-H",
        "orgId": "org-A",
        "updatedAt": "2020-01-05"
    },
    "_et": "ProgressRecord"
}

I'm happy with that shape, but it feels like a clumsy misuse of the Entity class, because it can't define the shape of attrs.

I can use another validator, just want to make sure I'm not missing anything here.

[conorw]

I think I see the problem, at least with the way that I'm trying to filter on that nested attribute. dynamodb-toolbox generates a dynamo query using the full path, like:

ExpressionAttributeNames: { ..., '#attr1': 'data.findMe' },
ExpressionAttributeValues: { ..., ':attr1': true },
FilterExpression: '#attr1 = :attr1',

but dynamo needs each step of the path to be broken down, more like this:

ExpressionAttributeNames: { ..., '#attr1': 'data', '#attr2': 'findMe' },
ExpressionAttributeValues: { ..., ':attr2': true },
FilterExpression: '#attr1.#attr2 = :attr2',

Am I just using this wrong? Is there another way to specify the path in the toolbox query args?

I am definitely seeing the same thing with v0.3.4 i.e. I cant query filter based on a nested attribute e.g.

filters.push({ attr: 'val.isClientVisible', eq: true});

[shellscape]

@jeremydaly coming up on a year later and this is definitely still a bug. what were the results of your poking at this?

For anyone running across this, the ExpressionAtrributeNames and such mentioned here https://github.com/jeremydaly/dynamodb-toolbox/issues/106#issuecomment-749779095 can be passed as the second parameter to scan so you can override what dynamodb-toolbox would want to generate erroneously.

[suankan]

Hi @lwillmeth, @jeremydaly,

Please correct my understanding of this AWS Dynamo doc which says:

Each primary key attribute must be a scalar (meaning that it can hold only a single value). The only data types allowed for primary key attributes are string, number, or binary.

So even if we try to create a Secondary Index using some Nested Attribute - it won't work because Secondary Index is basically an attribute(s) for querying the data, they are just alternative attributes to Primary Key attributes. But I guess the same Types limitation applies for both Primary Key and Secondary Index attributes - they both must be Scalar (String, Number or Binary).

So if I understand the above AWS Dynamo doc correctly, in Dynamo Tables it is not even possible to execute a get query by non-Scalar attribute. Which means that if our Table is like this:

{
    "sk": "abc",
    "pk": "test-a",
    "_et": "Foo",
    "data": {
        "findMe": true
    }
}

Then we won't be able to run a query like filters: [{ attr: 'data.findMe', eq: true }]. Just because we can query our table only by Primary Key attributes or Secondary Index attributes. Which could only be Scalars, not Nested.

At the same time it should be possible to set the data attribute type to String, and to put e.g. a serialised JSON string so that the table item will be like this:

{
    "sk": "abc",
    "pk": "test-a",
    "_et": "Foo",
    "data": "{\"findMe\":true}"
}

Then to make the data attribute a Primary Key or create a Secondary Index using it. Then to run query like filters: [{ attr: 'data', eq: "{\"findMe\":true}" }].

But obviously such workaround will be a misconception to the nature of Nested Attributes. And also it will put a burden on serialising/deserialising this attribute when dealing with it.

[salv0]

Incurring in this same problem with version 0.3.5

Trying to access a nested attribute in the form: filters: { attr: 'address.countryName', eq: 'Italy' }

but this doesn't work. Any help would be appreciated!

[aldex32]

Hi all,

I started last week to use this library hoping it is mature enough. I am facing the same issue with v0.3.5, and looks like an old issue not solved yet.

[darbio]

Hi all,

I started last week to use this library hoping it is mature enough. I am facing the same issue with v0.3.5, and looks like an old issue not solved yet.

There are alternatives that are at version 1+ if you don't need the specifics that this library provides.

I have used dynamodb-one-table, electro db and this lib. Each has its upsides and downsides.

The basic features in this library are pretty static but it does concern me that we still aren't at 1.0.0 and that there are a bunch of bugs that are >1 year old. These factors prevent me using it more extensively.

[wolfm89]

It seems as if this is still not working. I am trying to receive entities that have a certain map filed set like this:

const notifications = await context.entities.Notification.query(`USER#${userId}`, {
      beginsWith: 'NOTIFICATION#',
      attributes: ['id', 'data'],
      filters: [{ attr: 'data.journeyId', eq: journeyId }],
    })

But I do get this error:

Type '{ attr: "data.journeyId"; eq: any; }' is not assignable to type 'ConditionsOrFilters<"userId" | "type" | "id" | "created" | "modified" | "entity" | "timestamp" | "read" | "data">'. Types of property 'attr' are incompatible. Type '"data.journeyId"' is not assignable to type '"userId" | "type" | "id" | "created" | "modified" | "entity" | "timestamp" | "read" | "data" | undefined'.ts(2322)

This is my entity:

const Notification = new Entity({
  name: 'Notification',
  table: TrainPriceMonitorTable,
  attributes: {
    userId: { partitionKey: true, type: 'string', prefix: 'USER#' },
    id: {
      sortKey: true,
      type: 'string',
      prefix: 'NOTIFICATION#',
    },
    type: { type: 'string', required: true },
    timestamp: { type: 'string', required: true },
    read: { type: 'boolean', required: true, default: false },
    data: { type: 'map' },
  },
} as const);

Should this actually be working right now?