Bad Performance with querying database

[efazati]

Hi, I have document with bad performance. I used some profiler and you can see result

Tue Apr  4 13:02:36 2017    /tmp/profile.pstat

         2399000 function calls (2399641 primitive calls) in 4.152 seconds

   Ordered by: internal time
   List reduced from 148 to 100 due to restriction <100>


   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  104/704    2.225    0.021    7.144    0.010 /env/local/lib/python2.7/site-packages/mongoengine/base/fields.py:233(ListField.to_python)
  2379676    1.854    0.000    2.460    0.000 /env/local/lib/python2.7/site-packages/mongoengine/fields.py:62(StringField.to_python)
     8/40    0.031    0.004    7.180    0.179 /env/local/lib/python2.7/site-packages/mongoengine/base/document.py:539(_from_son)
        1    0.007    0.007    7.305    7.305 /env/local/lib/python2.7/site-packages/mongoengine/queryset/queryset.py:83(QuerySet._populate_cache)

I use pymongo for same query and load in 0.3 sec what is problem?

[efazati]

product_list =  Product.objects(store=self, is_deleted=False,
                            first_related__exists=False,
                            published=True)

vs

product_list = db.product.find(product_list._query)[:8]

is there any way to make better? I used mongoengine 8.2 upgraded to last version but nothing goes better

[wojcikstefan]

Hi @efazati, thank you for this detailed report! Unfortunately, current performance of MongoEngine is suboptimal, especially in terms of deserializing large documents. See https://github.com/MongoEngine/mongoengine/labels/Performance for a list of similar reports. I'll be working on this shortly, but I'm happy to accept help if you have any ideas about speeding things up.

[efazati]

@wojcikstefan I know its not the best solution but i handle somehow with external class i wrote https://github.com/efazati/MonRead/blob/master/monread.py

again i know its not the best way. but i think just for query, this method can be helpful