PynamoDB icon indicating copy to clipboard operation
PynamoDB copied to clipboard

Regression in test_connection_make_api_call__binary_attributes due to JMESPathTypeError since botocore 1.37.12

Open musicinmybrain opened this issue 8 months ago • 1 comments

$ python3 -m venv _e
$ . _e/bin/activate
(_e) $ pip install -e .[signals]
(_e) $ pip install -r requirements-dev.txt
(_e) $ pytest --ignore-glob='tests/integration/*'
[…]
============================================== FAILURES ==============================================
__________________________ test_connection_make_api_call__binary_attributes __________________________

send_mock = <MagicMock name='send' id='140285739890208'>

    @mock.patch('botocore.httpsession.URLLib3Session.send')
    def test_connection_make_api_call__binary_attributes(send_mock):
        binary_blob = b'\x00\xFF\x00\xFF'
        resp_text = json.dumps({
            UNPROCESSED_ITEMS: {
                'someTable': [{
                    'PutRequest': {
                        'Item': {
                            'name': {STRING: 'daniel'},
                            'picture': {BINARY: base64.b64encode(binary_blob).decode(DEFAULT_ENCODING)},
                        }
                    }
                }],
            }
        })

        resp = mock.Mock(
            spec=AWSResponse,
            status_code=200,
            headers={},
            content=resp_text.encode(),
        )

        send_mock.return_value = resp

>       resp = Connection()._make_api_call('BatchWriteItem', {})

tests/test_base_connection.py:1617:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pynamodb/connection/base.py:340: in _make_api_call
    return self.client._make_api_call(operation_name, operation_kwargs)
_e/lib64/python3.13/site-packages/botocore/context.py:124: in wrapper
    return func(*args, **kwargs)
_e/lib64/python3.13/site-packages/botocore/client.py:981: in _make_api_call
    ) = self._resolve_endpoint_ruleset(
_e/lib64/python3.13/site-packages/botocore/client.py:1125: in _resolve_endpoint_ruleset
    endpoint_info = self._ruleset_resolver.construct_endpoint(
_e/lib64/python3.13/site-packages/botocore/regions.py:501: in construct_endpoint
    provider_params = self._get_provider_params(
_e/lib64/python3.13/site-packages/botocore/regions.py:556: in _get_provider_params
    param_val = self._resolve_param_from_context(
_e/lib64/python3.13/site-packages/botocore/regions.py:585: in _resolve_param_from_context
    self._resolve_param_as_operation_context_param(
_e/lib64/python3.13/site-packages/botocore/regions.py:619: in _resolve_param_as_operation_context_param
    return jmespath.search(path, call_args)
_e/lib64/python3.13/site-packages/jmespath/__init__.py:12: in search
    return parser.Parser().parse(expression).search(data, options=options)
_e/lib64/python3.13/site-packages/jmespath/parser.py:509: in search
    result = interpreter.visit(self.parsed, value)
_e/lib64/python3.13/site-packages/jmespath/visitor.py:94: in visit
    return method(node, *args, **kwargs)
_e/lib64/python3.13/site-packages/jmespath/visitor.py:171: in visit_function_expression
    return self._functions.call_function(node['value'], resolved_args)
_e/lib64/python3.13/site-packages/jmespath/functions.py:80: in call_function
    self._validate_arguments(resolved_args, signature, function_name)
_e/lib64/python3.13/site-packages/jmespath/functions.py:91: in _validate_arguments
    return self._type_check(args, signature, function_name)
_e/lib64/python3.13/site-packages/jmespath/functions.py:97: in _type_check
    self._type_check_single(actual[i], allowed_types,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <jmespath.functions.Functions object at 0x7f96d015a0d0>, current = None, types = ['object']
function_name = 'keys'

    def _type_check_single(self, current, types, function_name):
        # Type checking involves checking the top level type,
        # and in the case of arrays, potentially checking the types
        # of each element.
        allowed_types, allowed_subtypes = self._get_allowed_pytypes(types)
        # We're not using isinstance() on purpose.
        # The type model for jmespath does not map
        # 1-1 with python types (booleans are considered
        # integers in python for example).
        actual_typename = type(current).__name__
        if actual_typename not in allowed_types:
>           raise exceptions.JMESPathTypeError(
                function_name, current,
                self._convert_to_jmespath_type(actual_typename), types)
                self._convert_to_jmespath_type(actual_typename), types)
E           jmespath.exceptions.JMESPathTypeError: In function keys(), invalid type for value: None, expected one of: ['object'], received: "null"

_e/lib64/python3.13/site-packages/jmespath/functions.py:111: JMESPathTypeError
[…]
====================================== short test summary info =======================================
FAILED tests/test_base_connection.py::test_connection_make_api_call__binary_attributes - jmespath.exceptions.JMESPathTypeError: In function keys(), invalid type for value: None, expected...
============================ 1 failed, 376 passed, 18 warnings in 11.19s =============================

Looking at the dependency changes for the first failing test build of the python-pynamodb package in Fedora Rawhide lead me to suspect this to be associated with a botocore update, and I was able to confirm that this started happening with botocore 1.37.12:

(_e) $ pip install 'botocore==1.37.12'
(_e) $ pytest --ignore-glob='tests/integration/*'
[…]
====================================== short test summary info =======================================
FAILED tests/test_base_connection.py::test_connection_make_api_call__binary_attributes - jmespath.exceptions.JMESPathTypeError: In function keys(), invalid type for value: None, expected...
============================ 1 failed, 376 passed, 18 warnings in 12.11s =============================
(_e) $ pip install 'botocore==1.37.11'
(_e) $ pytest --ignore-glob='tests/integration/*'
[…]
================================= 377 passed, 19 warnings in 11.62s ==================================

musicinmybrain avatar Apr 17 '25 13:04 musicinmybrain

Oh, again...

ikonst avatar Apr 17 '25 15:04 ikonst