ocpp.v201.datatypes fail validation when optional values are not provided

[MacDue]

trafficstars

For example, if you use class ComponentType (from ocpp.v201.datatypes)

Which looks like this:

@dataclass
class ComponentType:
    name: str
    instance: Optional[str] = None
    evse: Optional[EVSEType] = None

and create the instance:

ComponentType(name="EVSE")

If you use that instance somewhere within a payload, you'll get a ocpp.exceptions.ValidationError as ComponentType will be serialized as:

{ "name": "EVSE", "instance": None, "evse": None }

rather than the correct:

{ "name": "EVSE" }

I only used ComponentType as an example here, the same thing happens with all the data types with Optional[<type>] = None

I don't think there's any way to fix this by changing the data types (a dataclass will always include all the fields), I think there probably needs to be another step that removes the optional fields that are None when the dataclass is converted to JSON.

---

P.s. The SampledValueType is missing the measurand field.

[proelke]

Thats correct. At present the solution would be something like this:

from dataclasses import asdict
from ocpp. charge_point import remove_nones
from ocpp.v201.datatypes import ComponentType

component_type = ComponentType(name='EVSE')
component_type = remove_nones(asdict(component_type))

Admittedly not very elegant. The proper fix is to adjust remove_nones to handle this. I was working on a PR to do this but got a little sidetracked with other things.

[MacDue]

I think it's possible to be done without requiring you to call any additional functions on the class/instance, for example, in Python 3.8+ you could make an optional-aware asdict that uses the typing information within the dataclass to decide to remove the nones, and call that here: https://github.com/mobilityhouse/ocpp/blob/master/ocpp/messages.py#L232

import typing
from dataclasses import asdict

# Note: get_origin() and get_args() are Python 3.8+ 
# (though there's ways to do the same in older versions)
def is_optional(field):
  # typing.Optional[x] is an alias for typing.Union[x, None]
  return (typing.get_origin(field) is typing.Union
          and type(None) in typing.get_args(field))

# Proof of concept (would be a little more complex for nested dataclasses types) 
def optional_aware_asdict(dataclass_instance):
  result = asdict(dataclass_instance)
  # Only remove 'None' values that are marked as optional in the dataclass
  for field in dataclass_instance.__dataclass_fields__.values():
    if result[field.name] is None and is_optional(field.type):
      del result[field.name]
  return result

[OrangeTux]

Great idea @MacDue . For versions younger than Python 3.8 we could use the typing-extensions plugin.

[OrangeTux]

Here is a test that fails with the current implementation of ocpp.charge_point.remove_nones():

def test_remove_nones():
    from dataclasses import asdict
    from ocpp. charge_point import remove_nones
    from ocpp.v201.datatypes import ComponentType, GetVariableDataType, EVSEType, VariableType
    from ocpp.v201.call import GetVariablesPayload

    payload = GetVariablesPayload(
        get_variable_data=[
            GetVariableDataType(
                component=ComponentType(
                    name="Component",
                    evse=EVSEType(
                        id=1
                    ),
                ),
                variable=VariableType(
                    name="Variable"
                )
            )
        ]
    )
    call = snake_to_camel_case(asdict(payload))

    assert remove_nones(call) == {
        'getVariableData': [{
            'component': {
                'name': 'Component',
                'evse': {
                    'id': 1,
                },
            },
            'variable': {
                'name': 'Variable',
            }
        }]
    }

[OrangeTux]

I implemented a solution based on the ideas of @MacDue . You can find it here: https://github.com/mobilityhouse/ocpp/commit/c5d8d632b6a90256ec02414597eb3984a17a4c06

[MacDue]

def old_get_origin(type):
  return getattr(type, '__origin__', None)

def old_get_args(type):
  return getattr(type, '__args__', None)

def is_optional(field):
  # typing.Optional[x] is an alias for typing.Union[x, None]
  # get_origin/get_args with fallbacks for pre-python 3.8
  get_origin = getattr(typing, 'get_origin', old_get_origin)
  get_args = getattr(typing, 'get_args', old_get_args)
  return (get_origin(field) is typing.Union
          and type(None) in get_args(field))

Looks great :+1: I don't think the typing-extensions module is needed (and I'm not sure it adds the functions), however, above is a version of is_optional that works for python 3.6 (maybe earlier too, not tested) to python 3.9.

I've tested this on Python 3.6 and Python 3.9 on my PC

[proelke]

For what its worth, Python 3.6 goes EOL on 23 Dec 2021.

[MacDue]

The change is still needed for Python 3.7

[Jared-Newell-Mobility]

With Python 3.7 EOL was 2023-06-27, due to the time passed and that this appears to be resolved with commit c5d8d632b6a90256ec02414597eb3984a17a4c06 , I'll close this for now.

[MacDue]

The commit linked was never merged (it's still only on the improve-remove-optional-empty-fields branch, it seems)

[Jared-Newell-Mobility]

Yes I missed that, thank-you for highlighting it. I'll schedule this for review.

[Jared-Newell-Mobility]

Merged so will close