Crash with latest mypy and django-stubs

[matejsp]

Bug report

Error running latest django-stubs.

Traceback (most recent call last):
  File "mypy/checkexpr.py", line 4881, in accept
  File "mypy/nodes.py", line 1881, in accept
  File "mypy/checkexpr.py", line 429, in visit_call_expr
  File "mypy/checkexpr.py", line 549, in visit_call_expr_inner
  File "mypy/checkexpr.py", line 1209, in check_call_expr_with_callee_type
  File "mypy/checkexpr.py", line 1292, in check_call
  File "mypy/checkexpr.py", line 1502, in check_callable_call
  File "mypy/checkexpr.py", line 1049, in apply_function_plugin
  File "/Users/myuser/.virtualenvs/py311/lib/python3.11/site-packages/mypy_django_plugin/transformers/querysets.py", line 233, in extract_proper_type_queryset_annotate
    annotated_type = get_or_create_annotated_type(api, model_type, fields_dict=fields_dict)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/myuser/.virtualenvs/py311/lib/python3.11/site-packages/mypy_django_plugin/transformers/models.py", line 669, in get_or_create_annotated_type
    model_module_file = api.modules[model_module_name]  # type: ignore
                        ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
KeyError: 'django_stubs_ext'

It works using django-stubs==1.7.0 without django-stubs-ext installed.

What's wrong

Got crash with django-stubs.

How is that should be

It should not crash.

System information

• OS: MacOSX Ventura 12.2
• python version: 3.11.2 or 3.8.13
• django version: 3.2.18
• mypy version: 1.1.1
• django-stubs version: 1.16.0
• django-stubs-ext version: 0.8.0

[matejsp]

[mypy]
warn_return_any = True
warn_unused_configs = True
disallow_untyped_defs = False
ignore_missing_imports = True
# follow_imports = normal
follow_imports = skip
files = api,account,apps,common
plugins = mypy_django_plugin.main

[mypy.plugins.django-stubs]
DJANGO_SETTINGS_MODULE = settings

[mypy-common/user_history.*]
disallow_untyped_defs = True
# To allow introduction per bounded context
follow_imports = normal

[matejsp]

Also tested with: django-stubs==1.8.0 -> WORKS django-stubs==1.9.0 -> not working

[matejsp]

I upgraded to django-stubs==4.2.0 and django-stubs-ext==4.2.0 and it is still completely broken:

Traceback (most recent call last):
  File "mypy/checkexpr.py", line 4881, in accept
  File "mypy/nodes.py", line 1881, in accept
  File "mypy/checkexpr.py", line 429, in visit_call_expr
  File "mypy/checkexpr.py", line 549, in visit_call_expr_inner
  File "mypy/checkexpr.py", line 1209, in check_call_expr_with_callee_type
  File "mypy/checkexpr.py", line 1292, in check_call
  File "mypy/checkexpr.py", line 1502, in check_callable_call
  File "mypy/checkexpr.py", line 1049, in apply_function_plugin
  File "/Users/myuser/.virtualenvs/env311/lib/python3.11/site-packages/mypy_django_plugin/transformers/querysets.py", line 236, in extract_proper_type_queryset_annotate
    annotated_type = get_or_create_annotated_type(api, model_type, fields_dict=fields_dict)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/matej.spiller-muys/.virtualenvs/bitstamp311/lib/python3.11/site-packages/mypy_django_plugin/transformers/models.py", line 669, in get_or_create_annotated_type
    model_module_file = api.modules[model_module_name]  # type: ignore
                        ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
KeyError: 'django_stubs_ext'

Is there really no workaround for this problem?

[sobolevn]

Please, provide a reproducer. It clearly works for others :)

[matejsp]

Hm hard to reproduce but it crashes at line: qry = models.DocumentRevision.objects.annotate(

    obsolete_at_filter_expression = (Q(obsolete_at__gte=datetime.datetime.now()) | Q(obsolete_at__isnull=True),)  # type: ignore  # noqa: E501
    if 'show_obsolete' in filtering and filtering['show_obsolete']:
        obsolete_at_filter_expression = tuple()  # type: ignore
        del filtering['show_obsolete']
    qry = models.DocumentRevision.objects.annotate(
        max_modified_date=Max('document__documentcontent__modified_at')
    ).filter(*obsolete_at_filter_expression, **filtering)

[matejsp]

And another crash after I comment annotate in previous one:

        data = (
            models.SomeEvent.objects.values('exception_name').order_by().annotate(Count('exception_name'))
        )

Traceback (most recent call last):
  File "mypy/checkexpr.py", line 4881, in accept
  File "mypy/nodes.py", line 1881, in accept
  File "mypy/checkexpr.py", line 429, in visit_call_expr
  File "mypy/checkexpr.py", line 549, in visit_call_expr_inner
  File "mypy/checkexpr.py", line 1209, in check_call_expr_with_callee_type
  File "mypy/checkexpr.py", line 1292, in check_call
  File "mypy/checkexpr.py", line 1502, in check_callable_call
  File "mypy/checkexpr.py", line 1049, in apply_function_plugin
  File "/Users/myuser/.virtualenvs/py311/lib/python3.11/site-packages/mypy_django_plugin/transformers/querysets.py", line 277, in extract_proper_type_queryset_values
    assert isinstance(default_return_type, Instance)
AssertionError: 

[matejsp]

We use 4.1.9 Django now (different as in original bug, but same bug).

The key that is searched for type info couple of lines before in transfomers/models.py:

    annotated_typeinfo = helpers.lookup_fully_qualified_typeinfo(
        cast(TypeChecker, api), model_module_name + "." + type_name
    )

model_module_name + "." + type_name has a value:

"django_stubs_ext.WithAnnotations[models__DocumentRevision, TypedDict({'max_modified_date': Any})]"

And for some reason django_stubs_ext is missing in the list.

[realsuayip]

I can also confirm on Python 3.11.4 with

django-stubs==4.2.2
django-stubs-ext==4.2.2
mypy==1.4.1
django==4.2.2

If you disable mypy cache, it does not crash.

Here is a piece of offending code:

def get_relations(self, obj: WithAnnotations[User]) -> list[str]:
    return [name for name, exists in obj.rels.items() if exists]

[matejsp]

How did you disable mypy cache youself? If I disable it using official docs it still does not work.

django-stubs==4.2.3
django-stubs-ext==4.2.2
mypy==1.4.1
django==4.1.9

version: 1.4.1
Traceback (most recent call last):
  File "mypy/checkexpr.py", line 4961, in accept
  File "mypy/nodes.py", line 1901, in accept
  File "mypy/checkexpr.py", line 428, in visit_call_expr
  File "mypy/checkexpr.py", line 548, in visit_call_expr_inner
  File "mypy/checkexpr.py", line 1208, in check_call_expr_with_callee_type
  File "mypy/checkexpr.py", line 1291, in check_call
  File "mypy/checkexpr.py", line 1501, in check_callable_call
  File "mypy/checkexpr.py", line 1048, in apply_function_plugin
  File "/Users/myuser/.virtualenvs/venv311/lib/python3.11/site-packages/mypy_django_plugin/transformers/querysets.py", line 233, in extract_proper_type_queryset_annotate
    annotated_type = get_or_create_annotated_type(api, model_type, fields_dict=fields_dict)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/myuser/.virtualenvs/venv311/lib/python3.11/site-packages/mypy_django_plugin/transformers/models.py", line 666, in get_or_create_annotated_type
    model_module_file = api.modules[model_module_name]  # type: ignore
                        ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
KeyError: 'django_stubs_ext'
documents/services/document_revisions.py:329: : note: use --pdb to drop into pdb
(venv311) ➜  myuser mypy --cache-dir=/dev/null --show-traceback

[realsuayip]

I used --no-incremental

https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-no-incremental

Removing .mypy_cache manually and running again also works.

[matejsp]

Thank you for the fast answer. It seems my case is a bit different for some reason because it does not work in my case: mypy --cache-dir=/dev/null --show-traceback --no-incremental

version: 1.4.1
Traceback (most recent call last):
  File "mypy/checkexpr.py", line 4961, in accept
  File "mypy/nodes.py", line 1901, in accept
  File "mypy/checkexpr.py", line 428, in visit_call_expr
  File "mypy/checkexpr.py", line 548, in visit_call_expr_inner
  File "mypy/checkexpr.py", line 1208, in check_call_expr_with_callee_type
  File "mypy/checkexpr.py", line 1291, in check_call
  File "mypy/checkexpr.py", line 1501, in check_callable_call
  File "mypy/checkexpr.py", line 1048, in apply_function_plugin
  File "/Users/myuser/.virtualenvs/venv311/lib/python3.11/site-packages/mypy_django_plugin/transformers/querysets.py", line 233, in extract_proper_type_queryset_annotate
    annotated_type = get_or_create_annotated_type(api, model_type, fields_dict=fields_dict)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/myuser/.virtualenvs/venv311/lib/python3.11/site-packages/mypy_django_plugin/transformers/models.py", line 666, in get_or_create_annotated_type
    model_module_file = api.modules[model_module_name]  # type: ignore
                        ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
KeyError: 'django_stubs_ext'

[matejsp]

I have managed to reproduce the issue with the following sample: bug_report3.tar.gz

Inside is runme script

#!/bin/bash
set -eu -o pipefail

rm -rf .venv
rm -rf .mypy_cache

python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
mypy sample --show-traceback

I get:

(.venv) ➜  bug_report mypy sample
sample/document_revision.py:13: error: 'Settings' object has no attribute 'READ_ONLY_DB'  [misc]
sample/document_revision.py:19: error: 'Settings' object has no attribute 'READ_ONLY_DB'  [misc]
sample/document_revision.py:19: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
If this issue continues with mypy master, please report a bug at https://github.com/python/mypy/issues
version: 1.4.1
sample/document_revision.py:19: : note: please use --show-traceback to print a traceback when reporting a bug

First issue is with following imports https://github.com/typeddjango/django-stubs/issues/1614 and second is this one.

[matejsp]

@sobolevn I tried the patch with the reproducable in previous comment, but it does not work:(https://github.com/typeddjango/django-stubs/pull/1613/files#diff-e548f38fb43163048bb2a1c1a209654cbee3c97d5ca25cfe16ca0abc5065e7c6).

It seems to go: (.venv) ➜ bug_report mypy sample --show-traceback sample.document_revision -> added print(print(file.fullname)) not defined_model_classes -> print('not defined_model_classes')

Then I tried adding:

        defined_model_classes = self.django_context.model_modules.get(file.fullname)
        if not defined_model_classes:
            print('not defined_model_classes')
            return [self._new_dependency("django_stubs_ext")]

But still didn't work with the same error :(

If I change: [mypy] follow_imports = normal (instead of skip)

it starts working ...

[matejsp]

For me current workaround is to add to mypy.ini:

[mypy]
follow_imports = skip

[mypy-django_stubs_ext.*]
follow_imports = normal

[flaeppe]

#1727 should've removed 1 error, according to the traceback. And the crashing code was removed in #2319.

Due to that I'm closing this. Feel free to post a new issue with any errors/tracebacks you're encountering.