Separating django-filter from relay connection

[spockNinja]

Problem

Right now, the use of django-filter is tightly coupled with the use of the relay graphql format. I think it would be useful to separate the two so that filtering can be applied to a normal graphql list as well.

Solution

Pull out the django-filter logic from DjangoFilterConnectionField into something like DjangoFilterMixin. Then apply that mixin to DjangoFilterConnectionField and a new DjangoFilterField.

I've done something like this with success already. There's a lot of copy-pasta though, and I think others could benefit from it. Here's what we are currently using.

from graphene import Field
from graphene_django.filter.utils import (
    get_filtering_args_from_filterset,
    get_filterset_class
)

class DjangoFilterField(Field):

    def __init__(self, _type, fields=None, extra_filter_meta=None,
                 filterset_class=None, *args, **kwargs):

        _fields = _type._meta.filter_fields
        _model = _type._meta.model

        self.fields = fields or _fields
        meta = dict(model=_model, fields=self.fields)
        if extra_filter_meta:
            meta.update(extra_filter_meta)
        self.filterset_class = get_filterset_class(filterset_class, **meta)
        self.filtering_args = get_filtering_args_from_filterset(self.filterset_class, _type)
        kwargs.setdefault('args', {})
        kwargs['args'].update(self.filtering_args)
        super().__init__(List(_type), *args, **kwargs)

    @staticmethod
    def list_resolver(manager, filterset_class, filtering_args,
                      root, args, context, info):
        filter_kwargs = {k: v for k, v in args.items() if k in filtering_args}
        qs = manager.get_queryset()
        qs = filterset_class(data=filter_kwargs, queryset=qs).qs
        return qs

    def get_resolver(self, parent_resolver):
        return partial(self.list_resolver, self.type._meta.model._default_manager,
                       self.filterset_class, self.filtering_args)

@syrusakbary If this sounds like a good idea, I'd be happy to whip up the PR.

[NikosVlagoidis]

any news with that?

[jole78]

I for one think it's a great idea. I have for long wanted to use filtering on "normal" lists without the added overweight of forcing me to use connections, edges, nodes when I basically want to just filter a small list.

[eamigo86]

Hi @spockNinja , I try to use your code into my test project but got this error: "List have not attribute _meta", in:

 super().__init__(List(_type), *args, **kwargs)

When I pass:

 super().__init__(_type, *args, **kwargs)

I not got error, but list_resolver function return a QuerySet and I got other logical error when make a query, because expected a single object and not a list of them.

Is there another way to return a list of objects or define this line to avoid this error? By the way, thanks for the great idea!!!

[Eraldo]

I also like the idea and I confirm the issue @eamigo86 is experiencing. I would prefer doing without the edges => node syntax however I don't want to loose the great filtering features.

[spockNinja]

I have not tested the code in the description with graphene 2.0 and the corresponding update to graphene-django. Hopefully I'll get a chance to take another crack at it soon, as it seems there is a lot of interest in this functionality.

[zahir-koradia]

I haven't tested this thoroughly, but the follow seems to work with graphene 2.0.

class DjangoFilterField(Field):
    '''
    Custom field to use django-filter with graphene object types (without relay).
    '''

    def __init__(self, _type, fields=None, extra_filter_meta=None,
                 filterset_class=None, *args, **kwargs):
        _fields = _type._meta.filter_fields
        _model = _type._meta.model
        self.of_type = _type
        self.fields = fields or _fields
        meta = dict(model=_model, fields=self.fields)
        if extra_filter_meta:
            meta.update(extra_filter_meta)
        self.filterset_class = get_filterset_class(filterset_class, **meta)
        self.filtering_args = get_filtering_args_from_filterset(
            self.filterset_class, _type)
        kwargs.setdefault('args', {})
        kwargs['args'].update(self.filtering_args)
        super().__init__(List(_type), *args, **kwargs)

    @staticmethod
    def list_resolver(manager, filterset_class, filtering_args, root, info, *args, **kwargs):
        filter_kwargs = {k: v for k,
                         v in kwargs.items() if k in filtering_args}
        qs = manager.get_queryset()
        qs = filterset_class(data=filter_kwargs, queryset=qs).qs
        return qs

    def get_resolver(self, parent_resolver):
        return partial(self.list_resolver, self.of_type._meta.model._default_manager,
                       self.filterset_class, self.filtering_args)

Key changes I have made are:

1. Tracking type as self.type now returns List instead of the type of list it is.
2. Changing params passed list_resolver.

I hope this is useful @Eraldo @eamigo86.

[rikvanderkemp]

I stumbled on the exact same issue, it would be nice to use django filter without relay. Any status update on this? How would I use the example @zahir-koradia gave?

[helpse]

@spockNinja I'm trying to use something similar to this:

class User(DjangoObjectType):
    class Meta:
        model = auth_models.User
        filter_fields = ('email', )
        interfaces = (Node, )
        connection = UserConnection


class UserConnection(Connection):
    extra = graphene.String()

    class Meta:
        node = User

From my understanding, User node needs to be passed UserConnection on it's meta, and UserConnection needs to be passed User.

I'm getting a cross-reference here. Any help?

[helpse]

@eamigo86 how does it look to you?

[helpse]

Seems like this was solved here: https://github.com/graphql-python/graphene-django/issues/304

[helpse]

FYI:

class UserConnection(Connection):
    extra = graphene.String()

    class Meta:
        abstract = True

class User(DjangoObjectType):
    class Meta:
        model = auth_models.User
        filter_fields = ('email', )
        interfaces = (Node, )
        connection_class = UserConnection

class Query(graphene.ObjectType):
    users_connection = DjangoFilterConnectionField(User, where=UserWhereInput())

[ArnaudPel]

I feel the need to relate this issue to #274, which looks like a generalization.

[rcelha]

@spockNinja Did you have any feedback on whether your propose is valid? It would be awesome to have that baked in graphene-django (I really don't want to use relay).

I'd gladly help with the implementation

[eamigo86]

Hi @rcelha , I am currently working to get a new version updated and with many improvements, should be available for next week. I still do not understand why graphene-django only focuses on the implementation of graphql with Relay since Django can be used with any frontend framework

[rcelha]

@eamigo86 Me either. I thought that graphene-django was the right place for this kind of feature. Of course I understand that there are many concerns regarding adding new APIs. I might be wrong, but I haven't seen not even a discussion about it

[clintonb]

At a minimum the documentation should state the focus on Relay upfront. As someone coming from DRF, the documentation seems incomplete.

https://docs.graphene-python.org/projects/django/en/latest/filtering/

[calocan]

The documentation bit me as well. I spent a couple hours trying to get filtering to work without knowing it relied on Relay, which I have no need for. When I try to use @zahir-koradia 's code above I get errors about duplicate fields, although a normal List field works fine with the same code. Thanks to everyone working on this.

[phalt]

Currently rewriting the docs, so we can make sure we cover this there.

[BenjjinF]

I noticed that there's an issue with regards to exact id filtering too. Both GlobalIDFilter and GlobalIDFormField make use of from_global_id(); this will fail silently and return the whole list if given a regular model id.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mekhami]

@phalt Can we reopen this issue until it's resolved? It's a valid concern.

[axwalker]

@BenjjinF I'm experiencing this issue too with filtering on model IDs - did you find a solution?

[axwalker]

For anyone coming to this for graphene-django==2.8.0 who also needs to be able to filter on foreign key IDs, this is the minimum I was able to extract from graphene-django-extras:

def get_filterset_class(filterset_class, **meta):
    """Get the class to be used as the FilterSet"""
    if filterset_class:
        # If were given a FilterSet class, then set it up and
        # return it
        return setup_filterset(filterset_class)
    return custom_filterset_factory(**meta)


class GrapheneFilterSetMixin(BaseFilterSet):
    FILTER_DEFAULTS = FILTER_FOR_DBFIELD_DEFAULTS


def setup_filterset(filterset_class):
    """ Wrap a provided filterset in Graphene-specific functionality."""
    return type(
        "Graphene{}".format(filterset_class.__name__),
        (filterset_class, GrapheneFilterSetMixin),
        {},
    )


def custom_filterset_factory(model, filterset_base_class=FilterSet, **meta):
    """
        Create a filterset for the given model using the provided meta data
    """
    meta.update({"model": model, "exclude": []})
    meta_class = type(str("Meta"), (object,), meta)
    filterset = type(
        str("%sFilterSet" % model._meta.object_name),
        (filterset_base_class, GrapheneFilterSetMixin),
        {"Meta": meta_class},
    )
    return filterset


class DjangoFilterField(graphene.Field):
    """
    Custom field to use django-filter with graphene object types (without relay).

    See here for more detail: https://github.com/graphql-python/graphene-django/issues/206
    """

    def __init__(
        self, _type, *args, fields=None, extra_filter_meta=None, filterset_class=None, **kwargs
    ):
        _fields = _type._meta.fields
        _model = _type._meta.model
        self.of_type = _type
        self.fields = fields or _fields
        meta = dict(model=_model, fields=self.fields)
        if extra_filter_meta:
            meta.update(extra_filter_meta)
        filterset_class = filterset_class or _type._meta.filterset_class
        self.filterset_class = get_filterset_class(filterset_class, **meta)
        self.filtering_args = get_filtering_args_from_filterset(self.filterset_class, _type)
        kwargs.setdefault("args", {})
        kwargs["args"].update(self.filtering_args)
        super().__init__(graphene.List(_type), *args, **kwargs)

    @staticmethod
    def list_resolver(manager, filterset_class, filtering_args, root, info, *args, **kwargs):
        filter_kwargs = {k: v for k, v in kwargs.items() if k in filtering_args}
        qs = manager.get_queryset()
        qs = filterset_class(data=filter_kwargs, queryset=qs, request=info.context).qs
        return qs

    def get_resolver(self, parent_resolver):
        return partial(
            self.list_resolver,
            self.of_type._meta.model._default_manager,  # pylint: disable=protected-access
            self.filterset_class,
            self.filtering_args,
        )

I couldn't use graphene-django-extras directly because I was having some other issues with that - and I had some concerns that it hadn't been updated for a few months.

[liliserf]

@axwalker thanks for posting this! Can you provide a little more context on how you would use this to implement global filtering on a DjangoObjectType?

[axwalker]

Sure:

import graphene
from graphene_django import DjangoObjectType
from django_filters.filterset import FilterSet

class FooFilterSet(FilterSet):
    class Meta:
        model = models.Foo
        fields = ["bar", "baz"]  # fields you want to be able to filter on


class FooNode(DjangoObjectType):
    class Meta:
        model = models.Foo
        filterset_class = FooFilterSet


class Query(graphene.ObjectType):
    foos= DjangoFilterField(FooNode)

Then you can query foos with a filter, ie:

query {
  foos(bar: 'abc') {
    name
    whateverOtherField
  }
}

[liliserf]

Ah, gotcha. Thanks for clarifying. Unfortunately this doesn't account for my main use-case of filtering related objects.

[axwalker]

What is it exactly you need?

[mekhami]

@axwalker Given the following models:

class Book(models.Model):
    pass

class Movie(models.Model):
    pass

class Discussion(models.Model):
    book = models.ForeignKey(Book, related_name="book_discussions")
    movie = models.ForeignKey(Movie, related_name="movie_discussions")
    body = models.CharField()

and the following graphene types

class BookType(DjangoObjectType):
    class Meta:
        model = Book

class MovieType(DjangoObjectType):
    class Meta:
        model = Movie

class DiscussionType(DjangoObjectType):
    class Meta:
        model = Discussion
        filterset_class = DiscussionFilter

class DiscussionFilter(django_filter.FilterSet):
    body = django_filters.CharFilter(lookup_expr="icontains")

    class Meta:
        model = Discussion
        fields = ['body']

When I query a discussion on a book, I'd expect the filters to be available through the relation.

query Book {
  book(id=1) {
    bookDiscussions(body: "example text") {
       body
       id
    }
  }
}

But instead, no filters are available in the relationship. I'd have to override the attribute on BookType and MovieType to declare them as DjangoFilterField(DiscussionType) but having to do that from every direction that a discussion can be related is very tedious, considering I've already declared the filter_fields on the DiscussionType definition.

[nicam]

Would be great to get this!

[Cimmanuel]

Who properly implemented this?

[ccsv]

I like the idea if you can get rid of the "Node and Edge" from the query that would make it cleaner and get rid of the interface = (Node,) in the meta class of a node.

Also I found this filter solution using a resolver which can probably be rewritten as a Mixin

https://stackoverflow.com/questions/61174897/how-we-can-make-filtered-query-in-schema-py-in-graphene-django-project-without-u

[tisuela]

This would be great! We currently only have graphene-django-extras to provide this functionality.

[hugos94]

I noticed that there's an issue with regards to exact id filtering too. Both GlobalIDFilter and GlobalIDFormField make use of from_global_id(); this will fail silently and return the whole list if given a regular model id. @BenjjinF

I'm having the same problem. I need the filter to work only with global_id and generate an error when searching with the regular model id.

[hugos94]

I noticed that there's an issue with regards to exact id filtering too. Both GlobalIDFilter and GlobalIDFormField make use of from_global_id(); this will fail silently and return the whole list if given a regular model id. @BenjjinF

I'm having the same problem. I need the filter to work only with global_id and generate an error when searching with the regular model id.

I identified in the main class that was replaced, that the created filterset is tested as valid. If it is not valid, the function throws an exception. https://github.com/graphql-python/graphene-django/blob/608af578d4fc446b4ae452f5d41595c42ba389f4/graphene_django/filter/fields.py#L81-L102

So I changed it to the following:

@staticmethod
def list_resolver(manager, filterset_class, filtering_args, root, info, *args, **kwargs):
    filter_kwargs = {k: v for k, v in kwargs.items() if k in filtering_args}
    
    qs = manager.get_queryset()
    
    filterset = filterset_class(data=filter_kwargs, queryset=qs, request=info.context)
    if filterset.is_valid():
        return filterset.qs

    raise ValidationError(filterset.form.errors)

In this way, it returns an error when the regular model id is used.

[xykylikuf001]

Try this for non relay filterset connection:

class DjangoLimitOffsetFilterConnectionField(DjangoFilterConnectionField):
    def __init__(self, type, fields=None, order_by=None, extra_filter_meta=None, filterset_class=None, *args, **kwargs):
        self._type = type
        self._fields = fields
        self._provided_filterset_class = filterset_class
        self._filterset_class = None
        self._extra_filter_meta = extra_filter_meta
        self._base_args = None

        kwargs.setdefault("limit", graphene.Int())
        kwargs.setdefault("offset", graphene.Int(description="Query offset"))
        kwargs.setdefault("ordering", graphene.String(description="Query order"))

        super(DjangoLimitOffsetFilterConnectionField, self).__init__(
            type,
            *args,
            **kwargs
        )

    @property
    def type(self):
        class NodeConnection(PaginationConnection):
            total_count = graphene.Int()

            class Meta:
                node = self._type
                name = '{}NodeConnection'.format(self._type._meta.name)

            def resolve_total_count(self, info, **kwargs):
                return self.iterable.count()

        return NodeConnection

    @classmethod
    def connection_resolver(cls, resolver, connection, default_manager, queryset_resolver, max_limit,
                            enforce_first_or_last, root, info, **args):
        first = args.get("first")
        last = args.get("last")
        offset = args.get("offset")
        before = args.get("before")

        if enforce_first_or_last:
            assert first or last, (
                "You must provide a `first` or `last` value to properly paginate the `{}` connection."
            ).format(info.field_name)

        if max_limit:
            if first:
                assert first <= max_limit, (
                    "Requesting {} records on the `{}` connection exceeds the `first` limit of {} records."
                ).format(first, info.field_name, max_limit)
                args["first"] = min(first, max_limit)

            if last:
                assert last <= max_limit, (
                    "Requesting {} records on the `{}` connection exceeds the `last` limit of {} records."
                ).format(last, info.field_name, max_limit)
                args["last"] = min(last, max_limit)

        if offset is not None:
            assert before is None, (
                "You can't provide a `before` value at the same time as an `offset` value to properly paginate the `{}` connection."
            ).format(info.field_name)

        # eventually leads to DjangoObjectType's get_queryset (accepts queryset)
        # or a resolve_foo (does not accept queryset)
        iterable = resolver(root, info, **args)
        if iterable is None:
            iterable = default_manager
        # thus the iterable gets refiltered by resolve_queryset
        # but iterable might be promise
        iterable = queryset_resolver(connection, iterable, info, args)
        on_resolve = partial(
            cls.resolve_connection, connection, args, max_limit=max_limit, default_manager=default_manager
        )

        if Promise.is_thenable(iterable):
            return Promise.resolve(iterable).then(on_resolve)

        return on_resolve(iterable)

    @classmethod
    def resolve_connection(cls, connection, args, iterable, max_limit=None, default_manager=None):

        if iterable is None:
            iterable = cls.get_manager()

        iterable = maybe_queryset(iterable)

        if isinstance(iterable, QuerySet):
            if iterable.model.objects is not default_manager:
                default_queryset = maybe_queryset(default_manager)
                iterable = cls.merge_querysets(default_queryset, iterable)

            _len = iterable.count()
        else:
            _len = len(iterable)

        ordering = args.get("ordering")

        if ordering:
            iterable = connection_from_list_ordering(iterable, ordering)

        connection = connection_from_list_slice(
            iterable,
            args,
            connection_type=connection,
            pageinfo_type=PageInfoExtra,
        )
        connection.iterable = iterable
        connection.length = _len

        return connection



def connection_from_list_slice(
        list_slice, args=None, connection_type=None, pageinfo_type=None
):
    args = args or {}
    limit = args.get("limit", None)
    offset = args.get("offset", 0)

    if limit is None:
        return connection_type(
            results=list_slice,
            page_info=pageinfo_type(
                has_previous_page=False,
                has_next_page=False
            )
        )
    else:
        assert isinstance(limit, int), "Limit must be of type int"
        assert limit > 0, "Limit must be positive integer greater than 0"

        paginator = Paginator(list_slice, limit)
        _slice = list_slice[offset:(offset + limit)]

        page_num = math.ceil(offset / limit) + 1
        page_num = (
            paginator.num_pages
            if page_num > paginator.num_pages
            else page_num
        )
        page = paginator.page(page_num)

        return connection_type(
            results=_slice,
            page_info=pageinfo_type(
                has_previous_page=page.has_previous(),
                has_next_page=page.has_next()
            )
        )


def connection_from_list_ordering(items_list, ordering):
    field, order = ordering.split(',')

    order = '-' if order == 'asc' else ''
    field = re.sub(r'(?<!^)(?=[A-Z])', '_', field).lower()

    return items_list.order_by(f'{order}{field}')