Can't use `attributes` option for Scan queries

[Bingdom]

With the following schema:

const PokemonEntity = new Entity({
    name: 'Pokemon',

    schema: item({
        pokeId: string().key(),
        label: string().optional(),
        level: number(),
    }),

    table: pokeTable,
});

Then, attempting to scan:

pokeTable
    .build(ScanCommand)
    .entities(PokemonEntity)
    .options({
        filters: {
                PokemonEntity: {attr: 'level', eq: 100 },
            },
        },
        attributes: [
            'pokeId',
            'label',
        ],
    })
    .send();

Would yield the following error:

Missing required attribute for formatting: 'level'. undefined

If I remove attributes, then I would get an array as expected.

I can confirm that this happens on the latest version v2.5.0 and AWS SDK v3.823.0.

If I make the values optional, then the error disappears for that attribute, but I then get missing attributes for 'created' and so on.

[ThomasAribart]

@Bingdom Thanks for reaching out! Will have a look at it ASAP!

[ThomasAribart]

@Bingdom Sorry I couldn't reproduce it on my side. Here's my unit test:

import type { __MetadataBearer } from '@aws-sdk/client-dynamodb'
import { DynamoDBClient } from '@aws-sdk/client-dynamodb'
import { DynamoDBDocumentClient, ScanCommand as _ScanCommand } from '@aws-sdk/lib-dynamodb'
import type { AwsStub } from 'aws-sdk-client-mock'
import { mockClient } from 'aws-sdk-client-mock'

const dynamoDbClient = new DynamoDBClient()
const documentClient = DynamoDBDocumentClient.from(dynamoDbClient)
let documentClientMock: AwsStub<object, __MetadataBearer, unknown>

const pokeTable = new Table({
  name: 'poke-table',
  partitionKey: { type: 'string', name: 'pokeId' },
  documentClient
})

const PokemonEntity = new Entity({
  name: 'Pokemon',
  schema: item({
    pokeId: string().key(),
    label: string().optional(),
    level: number()
  }),
  table: pokeTable
})

documentClientMock.on(_ScanCommand).resolves({
  Items: [{ _et: 'Pokemon', pokeId: 'pokeId', label: 'label' }]
})

const command = TestTable.build(ScanCommand)
  .entities(PokemonEntity)
  .options({
    filters: {
      Pokemon: { attr: 'level', eq: 100 }
    },
    attributes: ['pokeId', 'label']
  })

console.log(command.params())

const { Items } = await command.send()

expect(Items).toStrictEqual([{ pokeId: 'pokeId', label: 'label' }])

The test is green which makes me think that there's another problem at hand here.

[Bingdom]

Thanks for getting back to me so quickly.

I can see I've used it in other areas of my project without any issues. The only difference I can see is that I am using binary types for that particular entity.

I'll set up a reproducible example tomorrow. I was able to work around the issue by building the scan command for the doc client. It did happen on 1.12, so I suspect it's been around for a while.

[Bingdom]

Ok, I found an issue, but it's not the same error I got from AWS. I did notice an inconsistency, so it's likely got to do with it.

import type { __MetadataBearer } from "@aws-sdk/client-dynamodb";
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import {
	DynamoDBDocumentClient,
	ScanCommand as _ScanCommand,
} from "@aws-sdk/lib-dynamodb";
import { mockClient } from "aws-sdk-client-mock";
import {
	item,
	Table,
	Entity,
	string,
	ScanCommand,
	number,
} from "dynamodb-toolbox";
import { expect, test } from "vitest";

const dynamoDbClient = new DynamoDBClient();
const documentClient = DynamoDBDocumentClient.from(dynamoDbClient);
let documentClientMock = mockClient(documentClient);

const pokemonTable = new Table({
	name: "pokemon",
	partitionKey: { name: "pokeId", type: "string" },
	documentClient: documentClient,
});

const entityProps = {
	timestamps: {
		created: {
			savedAs: "created",
			name: "created",
		},
		modified: {
			savedAs: "modified",
			name: "modified",
		},
	},
} as const;

const PokemonEntity = new Entity({
	name: "PokemonEntity",

	...entityProps,

	schema: item({
		pokeId: string().key(),

		label: string().optional(),
		buffs: string(),
		buffsCount: number(),
		createdOnDevice: string(),
	}),

	table: pokemonTable,
});

documentClientMock.on(_ScanCommand).resolves({
	Items: [
		{
			_et: "RecipeLog",
			pokeId: "d6a407f6-8469-4805-8998-0f4e8de7825c",
			buffs: "A1E204",
			buffsCount: 3,
			label: "abc",
			createdOnDevice: "2025-04-29 05:41:47.301903",
			created: "2025-04-29T05:58:17.325Z",
			modified: "2025-04-29T05:58:17.325Z",
		},
	],
});

const command = pokemonTable
	.build(ScanCommand)
	.entities(PokemonEntity)
	.options({
		filters: {
			PokemonEntity: {
				attr: "label",
				eq: "abc",
			},
		},
		attributes: ["buffsCount", "pokeId", "createdOnDevice"],
	});

console.log(command.params());

//const { Items } = await documentClient.send(new _ScanCommand(command.params()));
const { Items } = await command.send();

test("Check expected result", () => {
	expect(Items).toStrictEqual([
		{
			buffsCount: 3,
			pokeId: "d6a407f6-8469-4805-8998-0f4e8de7825c",
			createdOnDevice: "2025-04-29 05:41:47.301903",
		},
	]);
});

My Projection Expression is buffsCount, but I am still getting buffs returned. The same applies to the internal attributes. In my example, I've renamed to created, but I would get both createdOnDevice and created back.

When I send the command with just the documentClient, I noticed the entire object was returned. So when it's sent to AWS, the attributes are properly trimmed, and may not be accounted for in your tests?

[ThomasAribart]

@Bingdom Okay I found it!

The bug happens when one attribute is a prefix of the other (here buffs and buffsCount + created and createdOnDevice). My RegExp was invalid and returned the shorter attribute as projected as well 😱

Thanks for creating this issue 👍 Should be fixed by https://github.com/dynamodb-toolbox/dynamodb-toolbox/pull/1204 in v2.6.6 🙌

[Bingdom]

Testing the fix now, and I can confirm that the typing output shows the prefix of the other attributes. So there's a bit of an inconsistency between the typing and the actual output now 😀