commercetools-python-sdk icon indicating copy to clipboard operation
commercetools-python-sdk copied to clipboard

Published 20 hours ago • verified publisher icon labd

fix: mocking errors for /product-projections/search

Open pshiu opened this issue 1 year ago • 0 comments

Hi! Don't know if this project accepts PRs, but in case it does:

Description

On tests of commercetools-python-sdk SDK calls to Product Projection Search, we see error:

marshmallow.exceptions.ValidationError: {'filter': ['Unknown field.'], 'markmatchingvariants': ['Unknown field.']}

In the first 2 commits, this PR attempts to correct the schema and urls used in commercetools.testing.product_projections's ProductProjectionsBackend.search() to squash this error.

In the last 2 commits, this PR proposes minor changes to the coverage GitHub Action job to squash bugs from the breaking changes in tooling dependencies, such as one in actions/upload-artifact@3 to actions/upload-artifact@4 of not being able to upload artifacts with the same name.

Additional Information

See commit messages for detailed explanation of changes.

Full error logs that prompted change are below. (But see commit messages first.)

Output of error resolved by a5b5f273d259cfde32e1620c5f32c2b04d2264c8

Note in the full logs below how this mocker matches to to get_by_id() instead of search():

<...>
../../.virtualenvs/workarea/lib/python3.8/site-packages/commercetools/testing/product_projections.py:85: in get_by_id
    params = utils.parse_request_params(_ProductProjectionQuerySchema, request)
<...>

Full logs, starting with where search() is called:

% pytest
<...truncated...>
commerce_coordinator/apps/commercetools/clients.py:240: in get_product_variant_by_course_run
    results = self.base_client.product_projections.search(False, filter=f"variants.sku:\"{cr_id}\"").results
../../.virtualenvs/workarea/lib/python3.8/site-packages/commercetools/services/product_projections.py:294: in search
    return self._client._get(
../../.virtualenvs/workarea/lib/python3.8/site-packages/commercetools/client.py:36: in _get
    response = self._http_client.get(self._base_url + endpoint, params=params)
../../.virtualenvs/workarea/lib/python3.8/site-packages/requests/sessions.py:602: in get
    return self.request("GET", url, **kwargs)
../../.virtualenvs/workarea/lib/python3.8/site-packages/requests_oauthlib/oauth2_session.py:521: in request
    return super(OAuth2Session, self).request(
../../.virtualenvs/workarea/lib/python3.8/site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
../../.virtualenvs/workarea/lib/python3.8/site-packages/requests_mock/mocker.py:185: in _fake_send
    return _original_send(session, request, **kwargs)
../../.virtualenvs/workarea/lib/python3.8/site-packages/requests/sessions.py:703: in send
    r = adapter.send(request, **kwargs)
../../.virtualenvs/workarea/lib/python3.8/site-packages/requests_mock/adapter.py:248: in send
    resp = matcher(request)
../../.virtualenvs/workarea/lib/python3.8/site-packages/commercetools/testing/abstract.py:174: in _matcher
    response = callback(request, **path_match.groupdict())
../../.virtualenvs/workarea/lib/python3.8/site-packages/commercetools/testing/product_projections.py:85: in get_by_id
    params = utils.parse_request_params(_ProductProjectionQuerySchema, request)
../../.virtualenvs/workarea/lib/python3.8/site-packages/commercetools/testing/utils.py:32: in parse_request_params
    obj = schema().load(params)
../../.virtualenvs/workarea/lib/python3.8/site-packages/marshmallow/schema.py:722: in load
    return self._do_load(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_ProductProjectionQuerySchema(many=False)>
data = {'filter': 'variants.sku:"course-v1:michiganx+injurypreventionx+1t2021"', 'markmatchingvariants': 'false', 'predicate_var': {}}

    def _do_load(
        self,
        data: (
            typing.Mapping[str, typing.Any]
            | typing.Iterable[typing.Mapping[str, typing.Any]]
        ),
        *,
        many: bool | None = None,
        partial: bool | types.StrSequenceOrSet | None = None,
        unknown: str | None = None,
        postprocess: bool = True,
    ):
        """Deserialize `data`, returning the deserialized result.
        This method is private API.
    
        :param data: The data to deserialize.
        :param many: Whether to deserialize `data` as a collection. If `None`, the
            value for `self.many` is used.
        :param partial: Whether to validate required fields. If its
            value is an iterable, only fields listed in that iterable will be
            ignored will be allowed missing. If `True`, all fields will be allowed missing.
            If `None`, the value for `self.partial` is used.
        :param unknown: Whether to exclude, include, or raise an error for unknown
            fields in the data. Use `EXCLUDE`, `INCLUDE` or `RAISE`.
            If `None`, the value for `self.unknown` is used.
        :param postprocess: Whether to run post_load methods..
        :return: Deserialized data
        """
        error_store = ErrorStore()
        errors = {}  # type: dict[str, list[str]]
        many = self.many if many is None else bool(many)
        unknown = (
            self.unknown
            if unknown is None
            else validate_unknown_parameter_value(unknown)
        )
        if partial is None:
            partial = self.partial
        # Run preprocessors
        if self._has_processors(PRE_LOAD):
            try:
                processed_data = self._invoke_load_processors(
                    PRE_LOAD, data, many=many, original_data=data, partial=partial
                )
            except ValidationError as err:
                errors = err.normalized_messages()
                result = None  # type: list | dict | None
        else:
            processed_data = data
        if not errors:
            # Deserialize data
            result = self._deserialize(
                processed_data,
                error_store=error_store,
                many=many,
                partial=partial,
                unknown=unknown,
            )
            # Run field-level validation
            self._invoke_field_validators(
                error_store=error_store, data=result, many=many
            )
            # Run schema-level validation
            if self._has_processors(VALIDATES_SCHEMA):
                field_errors = bool(error_store.errors)
                self._invoke_schema_validators(
                    error_store=error_store,
                    pass_many=True,
                    data=result,
                    original_data=data,
                    many=many,
                    partial=partial,
                    field_errors=field_errors,
                )
                self._invoke_schema_validators(
                    error_store=error_store,
                    pass_many=False,
                    data=result,
                    original_data=data,
                    many=many,
                    partial=partial,
                    field_errors=field_errors,
                )
            errors = error_store.errors
            # Run post processors
            if not errors and postprocess and self._has_processors(POST_LOAD):
                try:
                    result = self._invoke_load_processors(
                        POST_LOAD,
                        result,
                        many=many,
                        original_data=data,
                        partial=partial,
                    )
                except ValidationError as err:
                    errors = err.normalized_messages()
        if errors:
            exc = ValidationError(errors, data=data, valid_data=result)
            self.handle_error(exc, data, many=many, partial=partial)
>           raise exc
E           marshmallow.exceptions.ValidationError: {'filter': ['Unknown field.'], 'markmatchingvariants': ['Unknown field.']}

../../.virtualenvs/workarea/lib/python3.8/site-packages/marshmallow/schema.py:909: ValidationError
Output of error resolved by 59c762312d71732d5b8de5e67767c6d056ab0e49

Note in full logs below that now call is to search():

<...>
../../.virtualenvs/workarea/lib/python3.8/site-packages/commercetools/testing/product_projections.py:57: in search
    params = utils.parse_request_params(_ProductProjectionQuerySchema, request)
<...>

but _ProductProjectionQuerySchema is used instead of _ProductProjectionSearchSchema:

<...>
../../.virtualenvs/workarea/lib/python3.8/site-packages/commercetools/testing/product_projections.py:57: in search
    params = utils.parse_request_params(_ProductProjectionQuerySchema, request)
<...>

Full logs:

% pytest
<...truncated...>
commerce_coordinator/apps/commercetools/clients.py:240: in get_product_variant_by_course_run
    results = self.base_client.product_projections.search(False, filter=f"variants.sku:\"{cr_id}\"").results
../../.virtualenvs/workarea/lib/python3.8/site-packages/commercetools/services/product_projections.py:294: in search
    return self._client._get(
../../.virtualenvs/workarea/lib/python3.8/site-packages/commercetools/client.py:36: in _get
    response = self._http_client.get(self._base_url + endpoint, params=params)
../../.virtualenvs/workarea/lib/python3.8/site-packages/requests/sessions.py:602: in get
    return self.request("GET", url, **kwargs)
../../.virtualenvs/workarea/lib/python3.8/site-packages/requests_oauthlib/oauth2_session.py:521: in request
    return super(OAuth2Session, self).request(
../../.virtualenvs/workarea/lib/python3.8/site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
../../.virtualenvs/workarea/lib/python3.8/site-packages/requests_mock/mocker.py:185: in _fake_send
    return _original_send(session, request, **kwargs)
../../.virtualenvs/workarea/lib/python3.8/site-packages/requests/sessions.py:703: in send
    r = adapter.send(request, **kwargs)
../../.virtualenvs/workarea/lib/python3.8/site-packages/requests_mock/adapter.py:248: in send
    resp = matcher(request)
../../.virtualenvs/workarea/lib/python3.8/site-packages/commercetools/testing/abstract.py:174: in _matcher
    response = callback(request, **path_match.groupdict())
../../.virtualenvs/workarea/lib/python3.8/site-packages/commercetools/testing/product_projections.py:57: in search
    params = utils.parse_request_params(_ProductProjectionQuerySchema, request)
../../.virtualenvs/workarea/lib/python3.8/site-packages/commercetools/testing/utils.py:32: in parse_request_params
    obj = schema().load(params)
../../.virtualenvs/workarea/lib/python3.8/site-packages/marshmallow/schema.py:722: in load
    return self._do_load(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_ProductProjectionQuerySchema(many=False)>
data = {'filter': 'variants.sku:"course-v1:michiganx+injurypreventionx+1t2021"', 'markmatchingvariants': 'false', 'predicate_var': {}}

    def _do_load(
        self,
        data: (
            typing.Mapping[str, typing.Any]
            | typing.Iterable[typing.Mapping[str, typing.Any]]
        ),
        *,
        many: bool | None = None,
        partial: bool | types.StrSequenceOrSet | None = None,
        unknown: str | None = None,
        postprocess: bool = True,
    ):
        """Deserialize `data`, returning the deserialized result.
        This method is private API.
    
        :param data: The data to deserialize.
        :param many: Whether to deserialize `data` as a collection. If `None`, the
            value for `self.many` is used.
        :param partial: Whether to validate required fields. If its
            value is an iterable, only fields listed in that iterable will be
            ignored will be allowed missing. If `True`, all fields will be allowed missing.
            If `None`, the value for `self.partial` is used.
        :param unknown: Whether to exclude, include, or raise an error for unknown
            fields in the data. Use `EXCLUDE`, `INCLUDE` or `RAISE`.
            If `None`, the value for `self.unknown` is used.
        :param postprocess: Whether to run post_load methods..
        :return: Deserialized data
        """
        error_store = ErrorStore()
        errors = {}  # type: dict[str, list[str]]
        many = self.many if many is None else bool(many)
        unknown = (
            self.unknown
            if unknown is None
            else validate_unknown_parameter_value(unknown)
        )
        if partial is None:
            partial = self.partial
        # Run preprocessors
        if self._has_processors(PRE_LOAD):
            try:
                processed_data = self._invoke_load_processors(
                    PRE_LOAD, data, many=many, original_data=data, partial=partial
                )
            except ValidationError as err:
                errors = err.normalized_messages()
                result = None  # type: list | dict | None
        else:
            processed_data = data
        if not errors:
            # Deserialize data
            result = self._deserialize(
                processed_data,
                error_store=error_store,
                many=many,
                partial=partial,
                unknown=unknown,
            )
            # Run field-level validation
            self._invoke_field_validators(
                error_store=error_store, data=result, many=many
            )
            # Run schema-level validation
            if self._has_processors(VALIDATES_SCHEMA):
                field_errors = bool(error_store.errors)
                self._invoke_schema_validators(
                    error_store=error_store,
                    pass_many=True,
                    data=result,
                    original_data=data,
                    many=many,
                    partial=partial,
                    field_errors=field_errors,
                )
                self._invoke_schema_validators(
                    error_store=error_store,
                    pass_many=False,
                    data=result,
                    original_data=data,
                    many=many,
                    partial=partial,
                    field_errors=field_errors,
                )
            errors = error_store.errors
            # Run post processors
            if not errors and postprocess and self._has_processors(POST_LOAD):
                try:
                    result = self._invoke_load_processors(
                        POST_LOAD,
                        result,
                        many=many,
                        original_data=data,
                        partial=partial,
                    )
                except ValidationError as err:
                    errors = err.normalized_messages()
        if errors:
            exc = ValidationError(errors, data=data, valid_data=result)
            self.handle_error(exc, data, many=many, partial=partial)
>           raise exc
E           marshmallow.exceptions.ValidationError: {'filter': ['Unknown field.'], 'markmatchingvariants': ['Unknown field.']}

../../.virtualenvs/workarea/lib/python3.8/site-packages/marshmallow/schema.py:909: ValidationError

Testing Information

We tested these changes by verifying that the test errors in https://github.com/edx/commerce-coordinator/pull/135 (see this comment) turned green on our locals when manually loading this branch of the SDK in our code.

We verified format/tests/coverage jobs ran successfully in https://github.com/edx/commercetools-python-sdk/pull/1.

pshiu avatar Dec 21 '23 20:12 pshiu