rest witchcraft doesn't seem to handle sqlalchemy.dialects.mysql.LONGBLOB fields

[ygbourhis]

I have a class like this one:

from sqlalchemy import CHAR, Column, ForeignKey, String, TIMESTAMP, Table, text
from sqlalchemy.dialects.mysql import INTEGER, LONGBLOB, LONGTEXT, SMALLINT

class CSITECONTENT(Base):
    __tablename__ = 'C_SITE_CONTENT'

    CONTENT_KEY = Column(String(50), primary_key=True)
    CONTENT_VAL = Column(LONGBLOB, nullable=False)
    MIME_TYPE = Column(String(50), nullable=False)
    CREATED_DATE = Column(TIMESTAMP, nullable=False, server_default=text("'0000-00-00 00:00:00'"))
    MODIFIED_DATE = Column(TIMESTAMP, nullable=False, server_default=text("'0000-00-00 00:00:00'"))

the issue is with sqlalchemy.dialects.mysql.LONGBLOB fields I get this kind of tracebacks:

Internal Server Error: /CSITECONTENT/
Traceback (most recent call last):
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_framework/viewsets.py", line 114, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_framework/views.py", line 505, in dispatch
    response = self.handle_exception(exc)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_framework/views.py", line 465, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
    raise exc
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_framework/views.py", line 502, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_framework/mixins.py", line 43, in list
    return self.get_paginated_response(serializer.data)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_framework/serializers.py", line 757, in data
    ret = super().data
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_framework/serializers.py", line 261, in data
    self._data = self.to_representation(self.instance)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_framework/serializers.py", line 675, in to_representation
    self.child.to_representation(item) for item in iterable
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_framework/serializers.py", line 675, in <listcomp>
    self.child.to_representation(item) for item in iterable
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/django_restql/mixins.py", line 77, in to_representation
    return super().to_representation(instance)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_framework/serializers.py", line 511, in to_representation
    for field in fields:
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_framework/serializers.py", line 372, in _readable_fields
    for field in self.fields.values():
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/django_restql/mixins.py", line 230, in fields
    return self.get_allowed_fields()
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/django_restql/mixins.py", line 80, in get_allowed_fields
    fields = super().fields
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/django/utils/functional.py", line 80, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_framework/serializers.py", line 360, in fields
    for key, value in self.get_fields().items():
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_witchcraft/serializers.py", line 373, in get_fields
    _fields[field_name] = self.build_field(source, info, self.model, depth)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_witchcraft/serializers.py", line 489, in build_field
    return self.build_standard_field(field_name, prop)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_witchcraft/serializers.py", line 100, in build_standard_field
    field_class = self.get_field_type(column_info)
  File "/home/yves/.virtualenvs/pycugate/lib/python3.6/site-packages/rest_witchcraft/serializers.py", line 114, in get_field_type
    "Could not figure out type for attribute '{}.{}'".format(self.model.__name__, column_info.property.key)
KeyError: "Could not figure out type for attribute 'CSITECONTENT.CONTENT_VAL'"

there seems to be an issue with LONGBLOB fields.

[ygbourhis]

For your information, the model was generated on a legacy db with sqlacodegen: https://pypi.org/project/sqlacodegen/ I only have read access to the DB and can not modify it.

[shosca]

Blob fields are not auto-mapped to file upload fields as they're not implemented yet. Though, you can try registering a file upload field or a custom serializer field that handles blobs by adding to rest_witchcraft.field_mapping.SERIALIZER_FIELD_MAPPING.

[ygbourhis]

Thanks @shosca for your reply.

file fields are different in django since they do not store the file data in the db, only the file access and flat files are kept for storage. The django equivalent of sqlalchemy.dialects.mysql.LONGBLOB fields is rather django.db.models.fields.BinaryField:

https://docs.djangoproject.com/en/3.0/ref/models/fields/#binaryfield

MySQL LONGBLOB type is for binary data up to 2^32 bytes.

They are used to store large data in the db.

[shosca]

Exactly, my plan was to do a simple Base64BinaryField that would dump/read base64 data from the blob columns but never got around to it

[shosca]

One thing that you can do for the api is to defer the content value, so something like this:

class SiteContentSerializer(ModelSerizlier):
    class Meta:
        model = SiteContent
       ...
        fields = ["key", "mime_type", "created_date", "updated_date"]


class SiteContentViewSet(ModelViewSet):
    ...
    serializer_class = SiteContentSerializer
    ...
    @action(methods=["get"], detail=True)
    def value(self):
        ... return the blob here ....

This way you will have content/<pk>/value giving you the blob

[ygbourhis]

@shosca for the moment I just replaced LONGBLOB by LONGTEXT(binary=True) and it seems to work. With LONGTEXT(binary=True) the ORM returns bytes instead of text. In fact this field can hold anything: text, png, etc... and with LONGTEXT(binary=True) I retrieve for PNG images b'\x89PNG\r\n\x1a\n\x00\x00\...' (etc...) If I then do an open('my_file', 'wb') and write the content, I get what I expect (a png if it was a png, etc...).