fix: assertion of DynamoType failing

[verysamuel]

This is a complete test which reproduces the issue.

The reload function is sourced from this Stack Overflow response. Our tests currently rely on this behaviour. The use of this function contributes to the occurrence of this bug.

import importlib
import sys

import boto3
import pytest
from boto3.dynamodb.types import TypeSerializer
from moto import mock_dynamodb

serializer = TypeSerializer()


@pytest.fixture(autouse=True)
def reload(monkeypatch):
    importlib.reload(sys.modules["moto"])
    to_reload = [m for m in sys.modules if m.startswith("moto.")]
    for m in to_reload:
        importlib.reload(sys.modules[m])


@pytest.fixture
def setup_dynamo():
    with mock_dynamodb():
        dynamodb = boto3.client("dynamodb", region_name="eu-west-1")
        dynamodb.create_table(
            TableName="SomeTable",
            AttributeDefinitions=[
                {"AttributeName": "SomeKey", "AttributeType": "S"},
            ],
            KeySchema=[
                {"AttributeName": "SomeKey", "KeyType": "HASH"},
            ],
            BillingMode="PAY_PER_REQUEST",
        )
        yield dynamodb


@pytest.mark.parametrize(
    "value",
    [
        serializer.serialize(value)
        for value in [
            {"This test case passes": "A"},
            {"This test case fails": "B"},
        ]
    ],
)
def test_reproduce_dynamo_type_assertion_error(setup_dynamo, value):
    config_key = "SomeKeyValue"

    setup_dynamo.put_item(
        TableName="SomeTable",
        Item={"SomeKey": {"S": config_key}, "ConfigValue": serializer.serialize({})},
    )

    setup_dynamo.update_item(
        TableName="SomeTable",
        Key={
            "SomeKey": {"S": config_key},
        },
        UpdateExpression="SET SomeValue = :some_value",
        ExpressionAttributeValues={":some_value": value},
    )

    dynamo_item = setup_dynamo.get_item(
        TableName="SomeTable",
        Key={"SomeKey": {"S": config_key}},
    )["Item"]["SomeValue"]

    assert dynamo_item == value

Running the above code results in this assertion failing: https://github.com/getmoto/moto/blob/bfac8a8a07f694dcbfa773c052dd7bf45d7fde06/moto/dynamodb/parsing/ast_nodes.py#L330

It seems something along the lines of this is going on: the reload function combined with the inconsistent imports means two different instances of DynamoType coexist and are in use.

Because the reload function is called between test invocations, this error only occurs when more than one test runs in a single pytest execution. So with the example above, running each case individually they'll both pass.

If appropriate, I can add the example above as a test here.

[verysamuel]

@bblommers "fun to debug!" is certainly one way to put it 😅

I have added a self-contained test which reproduces the issue using the style you've suggested.

[bblommers]

Hmm.. Our CI still doesn't seem to like that.

All the tests after the new tests are now failing:

2023-11-28T21:23:54.2345124Z tests/test_dynamodb/test_dynamodb.py::test_dynamo_type_assertion_exception_not_raised_when_reloading_modules PASSED
2023-11-28T21:23:54.3234042Z tests/test_dynamodb/test_dynamodb_batch_get_item.py::test_batch_items_returns_all PASSED
2023-11-28T21:23:54.7353450Z 
2023-11-28T21:23:54.7402019Z ##[error]test_batch_items_throws_exception_when_requesting_100_items_for_single_table

botocore.errorfactory.ResourceInUseException: An error occurred (ResourceInUseException) when calling the CreateTable operation: Table already exists: users
2023-11-28T21:23:54.7414803Z tests/test_dynamodb/test_dynamodb_batch_get_item.py::test_batch_items_throws_exception_when_requesting_100_items_for_single_table FAILED