Added draft DEP for external query languages support by Django ORM

[Nepherhotep]

Hey @jarshwah @akaariai

Here is a DEP describing the possible API for third-party query language support by Django ORM. If you like the idea it would be great if we can arrange phone/skype call to discuss some technical details, as the proposal touches code which I don't fully understand - basically it's stuff related to joins.

Best regards, Alexey

[jarshwah]

Hi @Nepherhotep

Have you thought about ExpressionBuilders being Resolvable rather than a separate build_expression method? Since Resolvable isn't really explained in the docs, it is simply a class that has a resolve_expression method that takes the arguments query=None, allow_joins=True, reuse=None, summarize=False, for_save=False and returns an Expression. The F class is a Resolvable, but isn't an expression itself.

Left right here - Django would not actually need to change any code at all. Since the library would have access to the current query object, it'd know the entire state of the query, have access to the model, and the Meta class of that model.

If above serves our purposes, then I think it'd be worth looking more closely into build_filter, and attempting to break it down into smaller helper methods that would allow a query library to extract the data it needs, and to push through the required joins. As you've written, Django is already doing the heavy lifting.

The ExpressionBuilder library would then maintain state internally of the property accesses and method calls, keeping an AST of sorts internally. For each property it'd be able to call query.model.get_field(field_name) for validation, and convert that into an F object (or a Col/Ref directly, skipping a further resolve step). Method calls it would do a lookup in the transforms dict. Finally, the last access would be represented by the operators, and would map to a lookup in the lookups dict.

A different library could also just convert the internal representation into a kwarg, and let django handle the resolving of each part - incurring more overhead but would seemingly be quicker to iterate on and get feedback for.

[schinckel]

I wonder if this might make the stuff @jarshwah and I have discussed before about allowing, for example, an Exists subquery to be used as a filter directly, instead of the annotate, filter dance we currently have to do.

[Nepherhotep]

I believe Resolvable will work, the critical thing here - ExpressionBuilder should not be of the whole Expression interface, as that class is quite complicated and it would be more difficult to extended it for third-party libraries (due to possible method conflicts, for example). As to transforms/lookups dicts, I'm not sure I fully understand. Once we manually create transforms/lookups hierarchy by QL, do we need to lookup them somewhere in query internal state at all? Assuming QL libraries will not encode transforms/lookups into field paths, I believe we don't. Also Django API should encapsulate join functionality inside and not delegate this job to third-party libraries.

[coady]

You might be interested in django-model-values' F expressions, as a reference implementation of essentially the same idea. The only difference is I choose to just subclass F, so the intermediate objects are themselves F expressions.

[jarshwah]

Cool library @coady! I'll definitely be taking a closer look.

[auvipy]

A mix of both approaches might be great

[auvipy]

Hi @Nepherhotep

Have you thought about ExpressionBuilders being Resolvable rather than a separate build_expression method? Since Resolvable isn't really explained in the docs, it is simply a class that has a resolve_expression method that takes the arguments query=None, allow_joins=True, reuse=None, summarize=False, for_save=False and returns an Expression. The F class is a Resolvable, but isn't an expression itself.

Left right here - Django would not actually need to change any code at all. Since the library would have access to the current query object, it'd know the entire state of the query, have access to the model, and the Meta class of that model.

If above serves our purposes, then I think it'd be worth looking more closely into build_filter, and attempting to break it down into smaller helper methods that would allow a query library to extract the data it needs, and to push through the required joins. As you've written, Django is already doing the heavy lifting.

The ExpressionBuilder library would then maintain state internally of the property accesses and method calls, keeping an AST of sorts internally. For each property it'd be able to call query.model.get_field(field_name) for validation, and convert that into an F object (or a Col/Ref directly, skipping a further resolve step). Method calls it would do a lookup in the transforms dict. Finally, the last access would be represented by the operators, and would map to a lookup in the lookups dict.

A different library could also just convert the internal representation into a kwarg, and let django handle the resolving of each part - incurring more overhead but would seemingly be quicker to iterate on and get feedback for.

I want to give this approach a shot you provide a green signal.

[auvipy]

we may also take https://github.com/thedrow/django-natural-query into consideration