Missing connection.queries

[ulgens]

After upgrading our project to django-debug-toolbar==4.1, we started having many test failures. Almost all of self.assertNumQueries(<query_count>) checks started failing saying the query count is different now. The queries we checked were actually the same as before, they were returning the same calls and returning the same data, but when I checked connection.queries, a majority of the queries were seemingly missed.

I did some testing with different commits and this problem starts with https://github.com/jazzband/django-debug-toolbar/commit/e7575e87dc9e2d2560b87d6fd5a123b9398cbd34

Not sure what else to add but if needed, I can try to create a small project reproducing this issue.

[tim-schilling]

Thank you for opening the issue. Creating a reproducible project would be very helpful.

[todorvelichkov]

We are facing kind of a similar issue, we have a logging.FileHandler that write all queries into a file, when DJDT is ON, no queries are being written into the file, however, the debug toolbar itself "sees" the queries (in the SQL tab).

The configuration is something like this:

LOGGING = {
    # ...
    'filters': {
        # ...
        'require_debug_true': {
            '()': 'django.utils.log.RequireDebugTrue'
        },
    },
    'handlers': {
        # ...
        'debug_console_to_file': {
            'level': 'DEBUG',
            'filters': [
                'require_debug_true'
            ],
            'class': 'logging.FileHandler',
            'filename': '/tmp/db.log'
        },
    },
    'loggers': {
        # ...
        'django.db.backends.schema': {
            'handlers': [
                'debug_console_to_file'
            ],
            'level': 'DEBUG',
            'propagate': False
        },
        'django.db.backends': {
            'handlers': [
                'debug_console_to_file'
            ],
            'level': 'DEBUG',
            'propagate': False
        },
        'django.db': {
            'handlers': [
                'debug_console_to_file'
            ],
            'level': 'DEBUG',
            'propagate': False
        },
    }
}

[tim-schilling]

@ulgens @todorvelichkov I can't investigate this without a test case or more information on how to reproduce this.

[ulgens]

@tim-schilling Sorry for the delay on my end. I'm no longer working on the project I encountered this issue. I'll try to create an example to reproduce this ~before the weekend~ at my first availability.

[bluetech]

Probably related to this report: since debug-toolbar 4.1 the django.db.backends SQL logger no longer works. I bisected this to commit e7575e87dc9e2d2560b87d6fd5a123b9398cbd34 "Inherit from django.db.backends.utils.CursorWrapper". From a quick look, the django.db.backends query logging is implemented in Django using a special CursorDebugWrapper that is instantiated instead of the normal CursorWrapper if settings.DEBUG is set:

https://github.com/django/django/blob/2584783f46922bcb456ceb9700a3726314df65d3/django/db/backends/base/base.py#L279-L288

I'm guessing something in the new commit now makes it use DJDT's cursor wrapper instead of the CursorDebugWrapper.

[akx]

@elnygren noted that this also breaks pytest-django's django_assert_num_queries and django.test.utils. CaptureQueriesContext.

[elnygren]

Also this person may be having the same problem https://github.com/pytest-dev/pytest-django/issues/1068

A really simple fix for testing is to INSTALLED_APPS.remove("debug_toolbar") which I guess makes sense to do anyway 😅