Groundwork for mypy support in FuzzManager

[nth10sd]

This PR lays the groundwork for mypy support in FuzzManager.

• In the misc/ folder, I had to rename afl-libfuzzer to afl_libfuzzer as mypy does not like dashes
• I took the mypy config in Lithium and adapted it here
• I added mypy and other types-* packages to the requirements3.0.txt file

GitHub Actions CI is here.

Current master branch has 3188 mypy errors after installing all the *-stubs packages (and their latest versions on a Python 3.10 venv), this PR now has 1558 errors left. (51.13% out of 3188 fixed)

[community-tc-integration[bot]]

No Taskcluster jobs started for this pull request

The `allowPullRequests` configuration for this repository (in `.taskcluster.yml` on the
default branch) does not allow starting tasks for this pull request.

[nth10sd]

(PR status updated below)

[nth10sd]

(PR status updated below)

[nth10sd]

(PR status updated below)

[nth10sd]

(PR status updated below)

[nth10sd]

(PR status updated below)

[nth10sd]

(PR status updated below)

[sylvestre]

Impressive work! bravo

[nth10sd]

Current master branch has 3188 mypy errors after installing all the *-stubs packages (and their latest versions on a Python 3.10 venv), this PR now has 1740 errors left. (45.42% out of 3188 fixed)

[nth10sd]

For now, CI is running on my fork.

[nth10sd]

~~Superseded by #829.~~

Oops, reinstating, I missed @jschwartzentruber 's PR from nth10sd/FuzzManager#1 into my fork.

[nth10sd]

@jschwartzentruber some questions:

1. Are you ok to import annotations now that we are on Python 3.7+? Or do you prefer old-style type annotations? (e.g. Optional[str])
2. For dictionaries, are you ok with dict[str, Any] typing?
3. For *args and **kwargs, are you ok with Any types?
4. For the items that I am not sure (e.g. decorator stuff), do you prefer me to put Any or you'd like to fill in the rest?
5. I still need to fix things up back to where I was, just prior to merging your changes in
6. ~~Another to-do, remove casting of models to int() and str()~~ (Done below)

This will take quite some time, and I have a work trip ahead that may take out the next few weeks for me.

Note to self, use tox -e mypy -- . and also use .[dev,server,taskmanager,docker,test] when installing via pip.

[jschwartzentruber]

1. Are you ok to import annotations now that we are on Python 3.7+? Or do you prefer old-style type annotations? (e.g. Optional[str])

yes, since we dropped 3.6 I think using the future import is fine.

2. For dictionaries, are you ok with dict[str, Any] typing?

yes

3. For *args and **kwargs, are you ok with Any types?

yes

4. For the items that I am not sure (e.g. decorator stuff), do you prefer me to put Any or you'd like to fill in the rest?

Any is fine

[nth10sd]

$ python -u -m tox -e mypy -- .
mypy develop-inst-noop: /home/skyxubu/FuzzManager
mypy installed: adal==1.2.7,aiohttp==3.8.1,aiosignal==1.2.0,amqp==2.6.1,apache-libcloud==2.4.0,appdirs==1.4.4,async-timeout==4.0.2,asynctest==0.13.0,attrs==21.4.0,azure-common==1.1.14,azure-mgmt-nspkg==3.0.2,azure-mgmt-resource==1.2.2,azure-nspkg==3.0.2,billiard==3.6.4.0,boto==2.49.0,boto3==1.21.3,botocore==1.24.3,celery==4.4.7,certifi==2021.10.8,cffi==1.15.0,charset-normalizer==2.0.12,coreapi==2.3.3,coreschema==0.0.4,coverage==6.3.1,cryptography==36.0.1,Deprecated==1.2.13,Django==2.2.27,django-chartjs==1.3,django-crispy-forms==1.11.2,django-enumfields==2.1.1,django-model-utils==4.2.0,django-notifications-hq==1.6.0,django-stubs==1.10.1,django-stubs-ext==0.4.0,djangorestframework==3.11.2,djangorestframework-stubs==1.5.0,fasteners==0.17.3,frozenlist==1.3.0,fuzzing-decision @ git+https://github.com/MozillaSecurity/orion@079c2eca4555998982f7f1459d252b2e047efc94#subdirectory=services/fuzzing-decision,-e git+ssh://[email protected]/nth10sd/FuzzManager.git@6593c254acd0686b444b0c66bb5ab478db98ae3b#egg=FuzzManager,idna==3.3,importlib-metadata==4.11.1,iniconfig==1.1.1,isodate==0.6.1,itypes==1.2.0,jeepney==0.7.1,Jinja2==3.1.2,jmespath==0.10.0,jsonfield==3.1.0,keyring==23.5.0,kombu==4.6.11,laniakea==1.17.1,MarkupSafe==2.1.1,mohawk==1.1.0,MozillaPulse==1.3,msrest==0.6.21,msrestazure==0.4.34,multidict==6.0.2,mypy==0.931,mypy-extensions==0.4.3,oauthlib==3.2.0,packaging==21.3,packet-python==1.37.1,pluggy==1.0.0,py==1.11.0,pycparser==2.21,pycryptodome==3.7.3,PyJWT==2.3.0,pyparsing==3.0.7,pytest==6.2.5,pytest-cov==3.0.0,pytest-django==4.5.2,pytest-mock==3.7.0,pytest-pythonpath==0.7.4,python-dateutil==2.8.2,pytz==2021.3,PyYAML==6.0,redis==4.1.4,requests==2.27.1,requests-oauthlib==1.3.1,s3transfer==0.5.1,SecretStorage==3.3.1,six==1.16.0,slugid==2.0.0,sqlparse==0.4.2,swapper==1.3.0,taskcluster==44.13.0,taskcluster-urls==13.0.1,toml==0.10.2,tomli==2.0.1,typed-ast==1.5.3,types-Markdown==3.3.14,types-pytz==2021.3.7,types-PyYAML==6.0.7,types-requests==2.27.25,types-urllib3==1.26.14,typing_extensions==4.1.1,uritemplate==4.1.1,urllib3==1.26.8,vine==1.3.0,whitenoise==5.2.0,wrapt==1.13.3,yarl==1.7.2,zipp==3.7.0
mypy run-test-pre: PYTHONHASHSEED='3039526982'
mypy run-test: commands[0] | mypy --install-types --non-interactive .
Error constructing plugin instance of NewSemanalDjangoPlugin

Traceback (most recent call last):
  File "/home/skyxubu/FuzzManager/.tox/mypy/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/home/skyxubu/FuzzManager/.tox/mypy/lib/python3.7/site-packages/mypy/__main__.py", line 12, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "mypy/main.py", line 96, in main
  File "mypy/main.py", line 173, in run_build
  File "mypy/build.py", line 180, in build
  File "mypy/build.py", line 231, in _build
  File "mypy/build.py", line 478, in load_plugins
  File "mypy/build.py", line 456, in load_plugins_from_config
  File "/home/skyxubu/FuzzManager/.tox/mypy/lib/python3.7/site-packages/mypy_django_plugin/main.py", line 70, in __init__
    self.django_context = DjangoContext(self.plugin_config.django_settings_module)
  File "/home/skyxubu/FuzzManager/.tox/mypy/lib/python3.7/site-packages/mypy_django_plugin/django/context.py", line 96, in __init__
    apps, settings = initialize_django(self.django_settings_module)
  File "/home/skyxubu/FuzzManager/.tox/mypy/lib/python3.7/site-packages/mypy_django_plugin/django/context.py", line 80, in initialize_django
    apps.populate(settings.INSTALLED_APPS)
  File "/home/skyxubu/FuzzManager/.tox/mypy/lib/python3.7/site-packages/django/apps/registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
  File "/home/skyxubu/FuzzManager/.tox/mypy/lib/python3.7/site-packages/django/apps/config.py", line 90, in create
    module = import_module(entry)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/skyxubu/FuzzManager/server/ec2spotmanager/__init__.py", line 3, in <module>
    from . import tasks  # noqa
  File "/home/skyxubu/FuzzManager/server/ec2spotmanager/tasks.py", line 15, in <module>
    from . import cron  # noqa ensure cron tasks get registered
  File "/home/skyxubu/FuzzManager/server/ec2spotmanager/cron.py", line 16, in <module>
    from .CloudProvider.CloudProvider import INSTANCE_STATE, PROVIDERS, CloudProvider
  File "/home/skyxubu/FuzzManager/server/ec2spotmanager/CloudProvider/CloudProvider.py", line 25, in <module>
    from ec2spotmanager.models import PoolConfiguration
  File "/home/skyxubu/FuzzManager/server/ec2spotmanager/models.py", line 44, in <module>
    class PoolConfiguration(models.Model):
  File "/home/skyxubu/FuzzManager/.tox/mypy/lib/python3.7/site-packages/django/db/models/base.py", line 103, in __new__
    app_config = apps.get_containing_app_config(module)
  File "/home/skyxubu/FuzzManager/.tox/mypy/lib/python3.7/site-packages/django/apps/registry.py", line 252, in get_containing_app_config
    self.check_apps_ready()
  File "/home/skyxubu/FuzzManager/.tox/mypy/lib/python3.7/site-packages/django/apps/registry.py", line 135, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
ERROR: InvocationError for command /home/skyxubu/FuzzManager/.tox/mypy/bin/mypy --install-types --non-interactive . (exited with code 1)
_________________________________________________________________________________ summary __________________________________________________________________________________
ERROR:   mypy: commands failed

@jschwartzentruber ~As requested by you, I switched to tox, and removed the casting, but I hit this error I cannot seem to fix. Do you mind unblocking this? (And also bumping mypy to the latest version while you're at it)~ I worked around this issue.

[nth10sd]

Note that the tox setup edits server/server/settings.py before running mypy, then reverts them to make mypy and its plugins happy. Suggestions welcome on how this should be resolved.

Before proceeding any further, @jschwartzentruber can you replicate this setup with tox -e mypy -- .? It'll be great if I can hand this off.