x-post to django-environ: https://github.com/joke2k/django-environ/issues/372

mypy --config-file ../mypy-django.ini project
Error constructing plugin instance of NewSemanalDjangoPlugin

Traceback (most recent call last):
  File "/home/sebastian/Repos/src/.tox/mypy/lib/python3.9/site-packages/environ/environ.py", line 367, in get_value
    value = self.ENVIRON[var]
  File "/home/sebastian/Repos/src/.tox/mypy/lib/python3.9/site-packages/environ/fileaware_mapping.py", line 43, in __getitem__
    return self.env[key]
  File "/usr/lib64/python3.9/os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'SECRET_KEY'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sebastian/Repos/src/.tox/mypy/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/home/sebastian/Repos/src/.tox/mypy/lib64/python3.9/site-packages/mypy/__main__.py", line 12, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "mypy/main.py", line 96, in main
  File "mypy/main.py", line 173, in run_build
  File "mypy/build.py", line 180, in build
  File "mypy/build.py", line 231, in _build
  File "mypy/build.py", line 478, in load_plugins
  File "mypy/build.py", line 456, in load_plugins_from_config
  File "/home/sebastian/Repos/src/.tox/mypy/lib/python3.9/site-packages/mypy_django_plugin/main.py", line 143, in __init__
    self.django_context = DjangoContext(django_settings_module)
  File "/home/sebastian/Repos/src/.tox/mypy/lib/python3.9/site-packages/mypy_django_plugin/django/context.py", line 95, in __init__
    apps, settings = initialize_django(self.django_settings_module)
  File "/home/sebastian/Repos/src/.tox/mypy/lib/python3.9/site-packages/mypy_django_plugin/django/context.py", line 77, in initialize_django
    settings._setup()
  File "/home/sebastian/Repos/src/.tox/mypy/lib/python3.9/site-packages/django/conf/__init__.py", line 74, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/sebastian/Repos/src/.tox/mypy/lib/python3.9/site-packages/django/conf/__init__.py", line 183, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/sebastian/Repos/src/project/project/settings.py", line 33, in <module>
    SECRET_KEY = env('SECRET_KEY')
  File "/home/sebastian/Repos/src/.tox/mypy/lib/python3.9/site-packages/environ/environ.py", line 175, in __call__
    return self.get_value(
  File "/home/sebastian/Repos/src/.tox/mypy/lib/python3.9/site-packages/environ/environ.py", line 371, in get_value
    raise ImproperlyConfigured(error_msg)
django.core.exceptions.ImproperlyConfigured: Set the SECRET_KEY environment variable
ERROR: InvocationError for command /home/sebastian/Repos/src/.tox/mypy/bin/mypy --config-file ../mypy-django.ini project (exited with code 1)

I get the approach of the django-stubs mypy plugin to execute the settings.py and I like the approach of django-environ to read settings from the environment.

How can we combine both?

My workaround right now looks like so:

if 'mypy_django_plugin' in sys.modules:
    # https://github.com/joke2k/django-environ/issues/372
    env = environ.Env(
        SECRET_KEY=(str, 'django-insecure'),
        DEBUG=(bool, False),
        DATABASE_URL=(str, 'sqlite:///sqlite.db'),
    )
else:
    env = ...

SECRET_KEY=xxx mypy --config-file ../mypy-django.ini project?

SECRET_KEY=xxx mypy --config-file ../mypy-django.ini project?

right, there are plenty of possible workarounds.

Unfortunately the issue is much much more deep here. When using mypy in pre-commit hooks, usually the isolated environment is created. In this case we HAVE TO add all applications used by django (specified in INSTALLED_APPS) inside additional_dependencies. Moreover we have to provide all required environment variables and sometimes it can be 100+.

The workaround for variables is to create specific mypy friendly settings file where you will override all the env variables using notation like:

import os

os.environ['SECRET_KEY'] = os.environ.get('SECRET_KEY', 'testing-key')

But for dependencies/requirements it is not the case. You have to specify all requirements in the mypy.ini or pyproject.toml file what make this file huge and difficult to maintain as you cannot reference your requirements file from there (at least by today)

What is the possible solution here? Any thoughts?

What is the possible solution here? Any thoughts?

I've found a couple of options for this, but it depends how you've organized your dev stack.

1. Write a script that calls mypy using your local virtualenv, then all the requirements will be available and you can avoid those unsightly duplicates. Here's a post describing that sort of approach which might help you out.

2. If you are running a dockerized stack you can just call out to the container and, assuming its got all the dev requirements installed on it, call mypy from inside that context. Put something like this in your pre-commit hook:

  - repo: local
    hooks:
    - id: mypy
      name: mypy
      entry: your-docker-image mypy
      language: docker_image
      require_serial: true
      'types_or': [python, pyi]
      verbose: true

Ensure your mypy.ini points to all the right places on the container and/not the local checkout.

Agree, we started to use also local hooks:

repos:
  - repo: local
    hooks:
      # Static Typing
      - args: ["run", "--", "--cache-fine-grained"]
        description: Static type checker for Python
        # https://github.com/pre-commit/mirrors-mypy/blob/main/.pre-commit-hooks.yaml
        entry: dmypy
        id: mypy
        language: system
        name: mypy
        require_serial: true
        types_or: [python, pyi]

We are using it not only for mypy but also for other tools like flake8, pylint, ruff, pyupgrade, vulture etc.

Why? Because we want to have these tools updated when dependabot updates the version and run linting or unit testing. Otherwise you have to manually update your pre-commit hooks and used version and validate that everything is working after upgrade. With local hooks and dependabot everything is happening naturally and automatically.

Best regards, Sergei