Explore Django Rest Framework extension using ModelSerializers

[pokken-magic]

I was thinking it'd be pretty feasible to use the DRF ModelSerializers to greatly simplify some complex tasks (related to issue #1375 ) like importing/exporting subsets of data with arbitrary depth.

So we could add a higher level of Resource that uses ModelSerializers or Serializers to import an model instance with its entire hierarchy of relationships intact, e.g. SerializerResource or the like.

It's entirely possible that Serializer could allow moving away from needing an underlying dataset engine, although I'm not sure if performance is there at scale or not. They seem pretty fast though.

(Possible that expanding to support another library in DRF is too big a scope, just an idea I had that might be worth some cycles)

[Hafnernuss]

I would very much welcome this for projects where DRF is already heavily used. I am facing this now. I have defined serializers for allmost all resources, including the creation/update of nested objects. If one could use those serializers for exporting/importing, this would greatly simplify the handling of nested objects.

[pokken-magic]

My thinking on the way to implement this would be to create a ModelSerializerResource that inherits from ModelResource, and overrides a lot of the functions. My recollection is that DRF has a way to find the default serializer for a given model, but you could also have a _meta option for specifying a specific serializer to use.

One challenge here is that I think it would only work cleanly for Json and Yaml, because CSV/Excel are very unreliable at nested data structures past a depth of one, unless the serializerfields are Json/yaml themselves.

The ModelSerializerResource would need to to override a lot of the functions on ModelResource and would probably not be able to use widgets and fields - because serializers essentially do what those do.

The first step would probably be to add DRF and a DRF app to the core test implementation, and then start fiddling with how serializers import. A really nice part about this is, at least as I recall, DRF serializers can handle creating related models for you, which includes an often-requested piece of functionality (creating related model instances).

Maybe I can spin up a branch for this one to do some brainstorming in the next couple months as I get some time.

[matthewhegarty]

It would help for issues like #1425, we could use the DRF DecimalField class to format and process numbers correctly. At the moment we are in danger of duplicating logic.

[pokken-magic]

If we're open to the possibility of adding DRF as a dependency I could see that being rather useful. I'm not sure architecturally speaking if we want to have DRF be required for import-export for people who are using vanilla django, but honestly DRF seems to be rapidly becoming the default for Django?

DRF's ModelSerializers do a lot of what import-export does natively so it'd probbly be a pretty major redesign though.

[Hafnernuss]

I don't know anything about the actual implementation of this library, but I've seen esupport libraries which add "DRF" capabilities to another library.

One small example is django-polymorphic and django-rest-polymorphic

But this would a.s. mean feature duplication.

[xblzbjs]

This is my solution. Hope it help.

core/mixins.py

from django.utils.translation import gettext_lazy as _

from import_export.formats.base_formats import CSV, HTML, JSON, ODS, TSV, XLS, XLSX, YAML
from import_export.resources import ModelResource
from rest_framework import status
from rest_framework.decorators import action
from rest_framework.response import Response
from tablib import Dataset

from .exceptions import ExportError


EXPORT_FORMATS_DICT = {
    "csv": CSV.CONTENT_TYPE,
    "xls": XLS.CONTENT_TYPE,
    "xlsx": XLSX.CONTENT_TYPE,
    "tsv": TSV.CONTENT_TYPE,
    "ods": ODS.CONTENT_TYPE,
    "yaml": YAML.CONTENT_TYPE,
    "json": JSON.CONTENT_TYPE,
    "html": HTML.CONTENT_TYPE,
}
IMPORT_FORMATS_DICT = EXPORT_FORMATS_DICT


class ExportMixin:
    """Export Mixin"""

    export_filename:str = "Default"
    export_resource: ModelResource = None

    @action(detail=False, methods=["get"])
    def export(self, request, *args, **kwargs):
        filename = self.export_filename
        eformat = request.query_params.get("eformat", "csv")
        
        queryset = self.filter_queryset(self.get_queryset())

        dataset = self.get_resource().export(queryset)

        if not hasattr(dataset, eformat):
            raise ExportError(
                detail=_("Unsupport export format"), code="unsupport_export_format"
            )

        data, content_type = (
            getattr(dataset, eformat),
            EXPORT_FORMATS_DICT[eformat],
        )

        response = HttpResponse(data, content_type=content_type)
        response["Content-Disposition"] = (
            f'attachment; filename="{filename}_{TODAY}.{eformat}"',
        )
        return response

    def get_resource(self):
        if not self.export_resource:
            raise ExportError(detail=_("Pleause set export resource"))
        return self.export_resource()


class ImportMixin:
    """Import Mixin"""

    import_resource: ModelResource = None

    @action(methods=["post"], detail=False)
    def import_data(self, request, *args, **kwargs):
        file = request.FILES["file"]
        extension = file.name.split(".")[-1].lower()
        import_resource = self.get_import_resource()
        dataset = Dataset()

        if extension in IMPORT_FORMATS_DICT:
            dataset.load(file.read(), format=extension)
        else:
            raise ImportError("Unsupport import format", code="unsupport_import_format")

        result = import_resource.import_data(
            dataset,
            dray_run=True,
            collect_failed_rows=True,
            raise_errors=True,
        )

        if not result.has_validation_errors() or result.has_errors():
            result = import_resource.import_data(
                dataset, dry_run=False, raise_errors=True
            )
        else:
            raise ImportError("Import data failed", code="import_data_failed")

        return Response(
            data={"message": "Import successed"}, status=status.HTTP_201_CREATED
        )

    def get_import_resource(self):
        if not self.import_resource:
            raise ImportError(detail=_("Pleause set import resource"))
        return self.import_resource()

core/exceptions.py

from rest_framework import exceptions


class ExportError(exceptions.APIException):
    status_code = 400
    default_detail = _("Export error")
    default_code = "export_error"


class ImportError(exceptions.APIException):
    status_code = 500
    default_detail = _("Import error")
    default_code = "import_error"