Reduce queries with prefetching

[jaydensmith]

Requests can easily blow out to hundreds of queries making it very slow. I'm not sure what the "best way" would be to solve this but I needed to fix it quickly and was able to go from 200 requests in a list of 50 objects to just 1.

Hopefully, there's some better way to have this built-in that would support fragments etc.

@strawberry.django.type(models.InventoryItem, pagination=True)
class InventoryItem:
    id: auto
    name: str
    category: 'InventoryCategory'
    supplier: 'InventorySupplier'

    def get_queryset(self, queryset, info, **kwargs):
        selection_set_node = info.field_nodes[0].selection_set
        fields = [selection.name.value for selection in selection_set_node.selections]
        related = [field for field in fields if field in ['category', 'supplier']]

        return queryset.select_related(*related)

[la4de]

This would be great improvement and I agree 100% with you. We really should do that. We probably would like to optimize the field fetching and query only fields which are actually requested by the end user.

[g-as]

The way I'm implementing it is the following:

class RootModel(models.Model):
    ...
    


class Model(models.Model):

    root_model = models.ForeignKey(RootModel)
    ...


class ReverseRelationModel(models.Model):

    root_model = models.ForeignKey(RootModel, related_name="reverse_relation_set")
    ...

@strawberry.django.type(ReverseRelationModel)
class ReverseRelationModelType:
    """ReverseRelationModel type."""

    root_model: "RootModelType"



@strawberry.django.type(RootModel)
class RootModelType:
    """RootModel type."""

    reverse_relation_set: List["ReverseRelationModelType"]



@strawberry.django.type(Model)
class ModelType:
    """Model type."""

    id: strawberry.django.auto
    root_model: RootModelType

    def get_queryset(  # pylint: disable=no-self-use
        self,
        queryset: QuerySet[Model],
        info: strawberry.types.Info,
        **_kwargs: T.Any,
    ) -> QuerySet[Model]:
        fields = get_selected_fields_from_info(info)

        if "rootModel" in fields:
            queryset = queryset.select_related("root_model")
            if "reverseRelationSet" in fields["rootModel"]:
                queryset = queryset.prefetch_related("root_model__reverse_relation_set")
        return queryset

from strawberry.types import Info
from strawberry.types.nodes import FragmentSpread, SelectedField, Selection


def get_selected_fields_from_info(info: Info) -> dict[str, dict]:
    """Fetch selected fields from fragments and ."""
    assert len(info.selected_fields) == 1
    return get_selected_fields_from_selections(info.selected_fields[0].selections)


def get_selected_fields_from_selections(selections: list[Selection]) -> dict[str, dict]:
    """Convert SelectedField's selection to dict of dicts."""
    res: dict[str, dict] = {}
    for sel in selections:
        if isinstance(sel, SelectedField):
            res.update({sel.name: get_selected_fields_from_selections(sel.selections)})
        elif isinstance(sel, FragmentSpread):
            res.update(get_selected_fields_from_selections(sel.selections))
    return res

[g-as]

But this feels hard to make generic. Especially if you need special prefetching cases using django's Prefetch.

[bellini666]

Hey guys. Just to point out that I created an optimizer extension in this lib here that does that and more: https://github.com/blb-ventures/strawberry-django-plus

edit: pasted the wrong link

[aareman]

@bellini666 does that library supersede this one, or that is an add-on to this library?

[bellini666]

@aareman it is basically an extension to this one, with some fixes, improvements and other new features (some that are still to come in the next days).

To use only the optimizer extension you would only need to add the extension itself to the schema and replace your strawberry.django.type and strawberry.django.field by my implementation. That implementation is a subclass of the one from here, but it makes sure that resolver querysets are optimized (i.e. using only, select_related, prefetch_related), and there's also an important optimization that avoids forcing the resolver to use sync_to_async when the model's values are already prefetched (and thus, won't hit the db).

[la4de]

Good job @bellini666 , It would be great to see all these improvements in core package one day.

[bellini666]

Hey @la4de , thanks! :). Yes, I'm planning on porting it to the core package, together with other improvements that make sense.

[hiporox]

@bellini666 Is there anything I can do to help porting that stuff over to the core package? All of those features are much needed

[bellini666]

Hey @hiporox . Help is always welcome, but this is something that I'm planning on incorporating to this project pretty soon (in the following weeks). I'm already very familiar with the code there and know some places I need to adjust here, so it should be pretty straight forward for me.

Also, I saw that you are sending some PRs and I'll review them all by the weekend. Thank you very much for helping the project :)

Having said all of that, if you want to use the optimizer right now while it is still not officially ported here, you can install my lib, import and use just the DjangoOptimizer extension and it should work out of the box for all places that you return a queryset.