mongoengine
mongoengine copied to clipboard
MongoEngine
Documents to json show all fields when using only() or exclude()
When converting a QuerySet with an only() or exclude() field to json, the required fields returned in each object are the required ones from the query.
output = Example.objects.only('field_1').to_json()
# Required feilds are returned
# [{'field_1': ...}, ...]
When converting a single document from a query with only paramaters the document containes the fields with default values.
output = Example.objects.only('field_1').first().to_json()
# All fields are returned with default values
# [{'field_1': ..., 'field_2': None, 'field_3': None, ...}, ...]
Do we have any update on this ticket? I'm facing this issue I need solution Please provide any update.
This is consistent with the docs: http://docs.mongoengine.org/guide/querying.html#retrieving-a-subset-of-fields
Apparently, "only()" just dictates what fields are effectively retrieved from the DB; but the fields are still filled with defaults when explicitly accesed
This might be consistent but still feels wrong. A DEFAULT value in a database object should be used only when writing to the database, in absence of a given value. Never for retrieve. It makes little sense (and it's very dangerous) that when I explicitly omit some attributes from the search, they are still returned, and with (very probably) wrong results.
The conceptual problem lies deeper in Mongoengie, as seen by the implementation of Document.delattr which, intends of deleting the attribute, fills it with the default value https://github.com/MongoEngine/mongoengine/blob/master/mongoengine/base/document.py That's not how delattr is supposed to work, generally speaking.
Any updates on this? I think this "bug" or "feature" is dangerous. Can easily be exploited for any purpose...
Any updates here? This makes the .only almost useless...
How can the desired behavior be to apply a default value that is frequently different than the actual value on a field? It cannot be that I am the only one that thinks filling fields with an incorrect value is wrong.
The problem here is what @leonbloy hit on. MongoEngine is operating off the underlying type of the Field class. If you check out the _from_son method, you'll see that when the document objects are instantiated, we're losing knowledge of what fields were included in the .only() projection and which weren't. It might be nice to record this information some where so that if to_json is called on the individual Document and not the Queryset, we know which fields to pluck.
I've been working around with the following code. Here's what I wrote for only_fields, which is used to apply the projection you used back onto the return dict from to_json.
By no means elegant, but I was not familiar enough with the codebase to figure out a better fix for the Field problem.
from projection import only_fields
# Example
user = User.objects.only("email").first()
correct_json = only_fields(user.to_json(), "email")
# projection.py
import re
DUNDER_SEPARATOR = "__"
PERIOD_SEPARATOR = "."
QUERY_SEPARATORS = (DUNDER_SEPARATOR, PERIOD_SEPARATOR)
_QUERY_SPLIT_REGEX = re.compile("|".join(re.escape(sep) for sep in QUERY_SEPARATORS))
def get_query_parts(key):
"""
Splits a MongoEngine-style query into an attribute sequence.
e.g. foo__bar -> ["foo", "bar"]
e.g. foo.bar -> ["foo", "bar"]
"""
parts = _QUERY_SPLIT_REGEX.split(key)
if parts and not parts[-1]:
# We have a "__" at the end that we need to handle
# e.g. foo__bar__type__ is a valid query string
return parts[:-1]
return parts
class DoNotInclude(Exception):
pass
def _list_startswith(prefix, array):
if len(array) < len(prefix):
return False
for i in range(len(prefix)):
if prefix[i] != array[i]:
return False
return True
def _only_fields_helper(obj, only_lookups, obj_lookup=None):
"""
Helper method for only_fields - do not call this directly.
:param obj: the object to inspect for projection
:param only_lookups: list of attribute sequences (see get_query_parts) to include in projection
:param obj_lookup: the attribute sequence of the current object
:return:
"""
obj_lookup = obj_lookup or []
if obj_lookup in only_lookups:
return obj
if not any(_list_startswith(prefix=obj_lookup, array=only_lookup) for only_lookup in only_lookups):
raise DoNotInclude
if isinstance(obj, dict):
newobj = {}
for k, v in obj.items():
try:
newobj[k] = _only_fields_helper(v, only_lookups, obj_lookup=obj_lookup + [k])
except DoNotInclude:
pass
return newobj
if isinstance(obj, list):
newobj = []
for el in obj:
try:
newobj.append(_only_fields_helper(el, only_lookups, obj_lookup=obj_lookup))
except DoNotInclude:
pass
return newobj
raise DoNotInclude
def only_fields(obj, *only):
"""
Plucks the specified fields from the given dict.
Note that you CANNOT specify indices in an array.
i.e. In MongoDB's find(query, projection), doing a projection for my_array__0 will not give you the first element
in the array - you will still get all elements back.
In this method, do not pass array indices to avoid making mistakes like this at all.
Example Usage:
obj = {my_array: [{foo: 1, bar: 2}, {foo: 3, bar: 4}]
only = ["my_array__foo"]
return = {my_array: [{foo: 1}, {foo: 3}]}
Bad Usage:
only = ["my_array__0__foo"] -> this will fail as array indices are not supported!
:param obj: dict to project
:param only: fields (specified in MongoEngine query format) to pluck
:return: dict of plucked fields
"""
if not only:
raise ValueError("At least one field must be specified in only.")
return _only_fields_helper(obj, [get_query_parts(el) for el in only])
