Is nested DynamicEmbeddedDocument of Mongoengine supported in DRF mongoengine?

[Isshwarya]

Hi,

More than a bug, this may be a new feature request. But please continue to read further and provide your guidance on how I can resolve it.

My model (a very lean and simplified version of what I actually have) looks like this:

class Job(DynamicDocument):
  name = StringField(max_length=64, required=True, unique=True)
  primary = EmbeddedDocumentField(PrimarySpec,  required=True)
  secondary = EmbeddedDocumentListField(SecondarySpec, required=True)

class PrimarySpec(DynamicEmbeddedDocument):
  name = StringField(max_length=64, required=True)
  spec = EmbeddedDocumentField(Spec, required=True)

class SecondarySpec(DynamicEmbeddedDocument):
  name = StringField(max_length=64, required=True)
  specs = EmbeddedDocumentListField(Spec, required=False)

class Spec(DynamicEmbeddedDocument):
  branch = StringField(max_length=64, required=True)

If I pass additional fields to primary.spec or secondary.specs (list), DynamicDocumentSerializer of rest_framework_mongoengine is failing to serialize those extra fields. It returns only the fields explicitly listed in model definition only.

Eg: I try to store this data in my model.

{
    "name": "foo",
    "primary": {
        "name": "abc",
        "spec": {
            "branch": "2.3.4", # This gets returned
            "version": "latest" # This is not returned by serializer
        }
    },
    "secondary": [
        {"name": "t1", "specs": [{"branch":"1.2.2", "version": "latest"}]}
    ]
}

Then I looked at the code of rest_framework_mongoengine/serializers.py to realize that we only have EmbeddedDocumentSerializer and DynamicDocumentSerializer but we dont have combination of both - DynamicEmbeddedDocumentSerializer

Then I attempted to make minor changes in rest_framework_mongoengine/serializers.py by defining

class DynamicEmbeddedDocumentSerializer(DynamicDocumentSerializer, EmbeddedDocumentSerializer):
  pass

and also I made this change just to try out:

def build_nested_embedded_field(self, field_name, relation_info, embedded_depth):
        #subclass = self.serializer_embedded_nested or EmbeddedDocumentSerializer
        subclass = self.serializer_embedded_nested or DynamicEmbeddedDocumentSerializer

Then it worked for primary.spec that I can specify fields that are not explicitly defined and the serializer returned those. But seconndary.specs didn't work even with all these changes (guess because it's list based and that might need additional changes in code and it's just that I could not figure out - I even checked the code for build_field() where it handles list or compound fields by recursing on child element types but my changes wasn't enough to handle the list type field though)

Please advise.

Thanks.

[IATF]

这是来自QQ邮箱的假期自动回复邮件。   您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。

[Isshwarya]

I could finally get it working with these derived classes. All the serializer classes corresponding to my models would inherit the class CustomDynamicDocumentSerializer

from rest_framework import fields as drf_fields, serializers as drf_serializers
from rest_framework_mongoengine import serializers as mongoserializers
from rest_framework_mongoengine.serializers import EmbeddedDocumentSerializer, DynamicDocumentSerializer
from rest_framework_mongoengine.utils import get_nested_embedded_kwargs

class CommonMixin(object):
  """
  Common mixin class. Since the common base class DynamicDocumentSerializer
  cannot be modified, this mixin class is used to share common logic
  """
  def to_internal_value(self, data):
    # This overridden implementation handles EmbeddedDocumentListField field
    # which DocumentSerializer (grandparent) class misses to do. That's the reason
    # for directly calling into DRF's Modelserializer's to_internal_value()
    # implementation skipping the inheritance hierarchy
    # for EmbeddedDocumentSerializers create initial data
    # so that _get_dynamic_data could use them
    for field in self._writable_fields:
      if isinstance(field, EmbeddedDocumentSerializer) and field.field_name in data:
        field.initial_data = data[field.field_name]
      if isinstance(field, drf_fields.ListField) and isinstance(field.child, EmbeddedDocumentSerializer):
        for entry in data[field.field_name]:
          field.child.initial_data = entry

    ret = drf_serializers.ModelSerializer.to_internal_value(self, data)

    # for EmbeddedDcoumentSerializers create _validated_data
    # so that create()/update() could use them
    for field in self._writable_fields:
      if isinstance(field, EmbeddedDocumentSerializer) and field.field_name in ret:
        field._validated_data = ret[field.field_name]
      if isinstance(field, drf_fields.ListField) and isinstance(field.child, EmbeddedDocumentSerializer):
        for entry in ret[field.field_name]:
          field.child._validated_data = entry

    dynamic_data = self._get_dynamic_data(ret)
    ret.update(dynamic_data)
    return ret

# Metaclass defined to set the same class object as value to its class attribute
def CustomMeta(name, bases, attrs):
    cls = type(name, bases, attrs)
    # This was needed to guide "serializer_embedded_nested" class decision
    # for nested EmbeddedDocument fields too. DocumentSerializer class uses
    # a private class called "EmbeddedSerializer" for embedded document fields.
    # That inherits DynamicEmbeddedDocumentSerializer because of the way we guided
    # through serializer_embedded_nested in CustomDynamicDocumentSerializer. But for
    # further nestings, this class should know what should  be used for 
    # serializer_embedded_nested attribute and that's the same class itself.
    cls.serializer_embedded_nested = cls
    return cls

# Inherits mixin and both the classes to get the combined behavior of "dynamic"
# as well as "embedded"
class DynamicEmbeddedDocumentSerializer(CommonMixin, DynamicDocumentSerializer, EmbeddedDocumentSerializer):
  __metaclass__ = CustomMeta
  
class CustomDynamicDocumentSerializer(CommonMixin, DynamicDocumentSerializer):
  """
  custom serializer that houses custom serialization logic.
  """
  # This specifies the class to be used to deal with EmbeddedDocument field.
  serializer_embedded_nested = DynamicEmbeddedDocumentSerializer

  ... <more custom stuff>...

I'm not aware of all possible scenarios that DRF mongoengine needs to consider to support Nested Dynamic Embedded documents. But this works for my case. I'm ready to work on this further and I can raise a pull request with all these changes if you suggest so.