Django mysqlclient issue with using Python 3.7.2

[sookyeomKim]

Context

I am trying to deploy a django app using mysql. However, an issue arises.

Using sqlite works fine. But, using mysql causes an error that fails to load MySQLdb.

There is nothing wrong with python3.6

An issue occurs in python3.7.

Actual Behavior

get a 502 error from Zappa API

Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.

The 'zappa tail' looks like this:

[1553164824234] [ERROR] ImproperlyConfigured: Error loading MySQLdb module.     ) from errask/django/db/backends/mysql/base.py", line 20, in <module>rt_module

Possible Fix

Support for python3.7 in lambda-packages(mysqlclient, etc...)

Steps to Reproduce

1. Deploy Zappa app with using mysqlclient

Your Environment

• Zappa version used: 0.48.2
• Operating System and Python version: mac and 3.7.2
• The output of pip freeze: argcomplete==1.9.3 boto3==1.9.118 botocore==1.12.118 certifi==2019.3.9 cfn-flip==1.1.0.post1 chardet==3.0.4 Click==7.0 Django==2.1.7 django-cors-headers==2.5.2 django-storages==1.7.1 djangorestframework==3.9.2 djangorestframework-stubs==0.3.0 docutils==0.14 durationpy==0.5 future==0.16.0 hjson==3.0.1 idna==2.8 jmespath==0.9.3 kappa==0.6.0 lambda-packages==0.20.0 mysqlclient==1.3.12 placebo==0.9.0 python-dateutil==2.6.1 python-decouple==3.1 python-slugify==1.2.4 pytz==2018.9 PyYAML==5.1 requests==2.21.0 s3transfer==0.2.0 six==1.12.0 toml==0.10.0 tqdm==4.19.1 troposphere==2.4.5 Unidecode==1.0.23 urllib3==1.24.1 Werkzeug==0.15.0 wsgi-request-logger==0.4.6 zappa==0.48.2
• Your zappa_settings.py: { "prod": { "aws_region": "ap-northeast-2", "django_settings": "LcBot.settings", "profile_name": "default", "project_name": "lc-bot", "runtime": "python3.7", "s3_bucket": "lc-bot" } }

[corwin-cole]

Having the same exact issue. In previous versions, a precompiled mysqlclient version (1.3.12) would deploy. The error "Failed to find library...right filename?" is printed before the actual raise. Thrown by this block in zappa/handler.py:

# Load compiled library to the PythonPath
# checks if we are the slim_handler since this is not needed otherwise
# https://github.com/Miserlou/Zappa/issues/776
is_slim_handler = getattr(self.settings, 'SLIM_HANDLER', False)
if is_slim_handler:
    included_libraries = getattr(self.settings, 'INCLUDE', ['libmysqlclient.so.18'])
    try:
        from ctypes import cdll, util
        for library in included_libraries:
            try:
                cdll.LoadLibrary(os.path.join(os.getcwd(), library))
            except OSError:
                print ("Failed to find library...right filename?")
    except ImportError:
        print ("Failed to import cytpes library")

Not sure if there's a package name that can be put into include or something to fix?

[jachicao]

Same issue for a Django app with mysqlclient and python3.7, here is the log from CloudWatch:

[ERROR] ImproperlyConfigured: Error loading MySQLdb module.

Failed to find library...right filename?

Did you install mysqlclient?
Traceback (most recent call last):
  File "/var/task/handler.py", line 602, in lambda_handler
    return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 245, in lambda_handler
    handler = cls()
  File "/var/task/handler.py", line 151, in __init__
    wsgi_app_function = get_django_wsgi(self.settings.DJANGO_SETTINGS)
  File "/var/task/zappa/ext/django_zappa.py", line 20, in get_django_wsgi
    return get_wsgi_application()
  File "/tmp/backend/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/tmp/backend/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/tmp/backend/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/tmp/backend/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/var/lang/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/backend/django/contrib/auth/models.py", line 2, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/tmp/backend/django/contrib/auth/base_user.py", line 47, in <module>
    class AbstractBaseUser(models.Model):
  File "/tmp/backend/django/db/models/base.py", line 117, in __new__
    new_class.add_to_class('_meta', Options(meta, app_label))
  File "/tmp/backend/django/db/models/base.py", line 321, in add_to_class
    value.contribute_to_class(cls, name)
  File "/tmp/backend/django/db/models/options.py", line 204, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/tmp/backend/django/db/__init__.py", line 28, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/tmp/backend/django/db/utils.py", line 201, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/tmp/backend/django/db/utils.py", line 110, in load_backend
    return import_module('%s.base' % backend_name)
  File "/var/lang/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/tmp/backend/django/db/backends/mysql/base.py", line 20, in <module>
    ) from err

[jtszalay]

EDIT: I was experiencing this issue: https://github.com/Miserlou/Zappa/issues/744

I changed my zappa lib locally to have the line

print ("Failed to find library {} ...right filename?").format(library)

instead of

print ("Failed to find library ...right filename?")

This prints the following on python 3.7.3: Failed to find library libmysqlclient.so.18 ...right filename?

I'm attempting to resolve this by adding the following to my config: "include":[]

Edit: looks like if you set the value to empy it doesn't override the search for libmysqlclient.so.18 Is there a way to make this not a default include?

[yangga]

downgrade Django==2.0.0

it works for me

[jbasko]

No, downgrading Django to 2.0 or 2.1 doesn't make a difference. Isn't this happening because there's no mysqlclient built for Python 3.7 in https://github.com/Miserlou/lambda-packages ?

Is Zappa still being developed? I'm just starting to use it today, but this is a blocker and now I see there are hundreds of issues open.

[shpyr]

Coming up on a year later and I'm seeing this still. Is there really no work-around yet?

[Kelvin97lui]

No, downgrading Django to 2.0 or 2.1 doesn't make a difference. Isn't this happening because there's no mysqlclient built for Python 3.7 in https://github.com/Miserlou/lambda-packages ?

Is Zappa still being developed? I'm just starting to use it today, but this is a blocker and now I see there are hundreds of issues open.

The same issue arises even with lower versions of Python 3.x

[ge0rg]

You can work around the lack of native binaries by using pymysql instead of mysqlclient as described in this SO post:

pip install pymysql

And add to your project's __init__.py:

import pymysql
pymysql.install_as_MySQLdb()

This will add about 6MB to your ZIP size.

[Kelvin97lui]

I installed pymysql and got the same error that @ox got in the SO post you linked.

In the SO post, user ox commented:

ERROR: django.core.exceptions.ImproperlyConfigured: mysqlclient 1.3.13 or newer is required; you have 0.9.3

However, I have also installed a version of mysqlclient 1.3.13 or newer.

[shpyr]

@Kelvin97lui it's pretty easy to get around that, here's what I ended up doing:

import pymysql
pymysql.install_as_MySQLdb()
pymysql.version_info = (1, 3, 13, 'final', 0)

^ this is at the top of my settings.py.

[glw]

Im getting this same error but not with a Django app. Im using Flask, but Im not using mysql for any part of my app. does zappa use this library in the background for anything? And should it be a part of zappa requirements?

[LyleScott]

Yeah, try "include": [],

This seems to knock out it out of the includes that zappa wants to auto generate with the code:

https://github.com/Miserlou/Zappa/blob/ba20c850eeca00edd6ea39fda1ab976cfee193ea/zappa/handler.py#L105

[MexsonFernandes]

import pymysql pymysql.install_as_MySQLdb() pymysql.version_info = (1, 3, 13, 'final', 0)

this one worked for me.

Django==2.0.0 Zappa==0.48.2

Python 3.6

[jottenlips]

import pymysql pymysql.install_as_MySQLdb() pymysql.version_info = (1, 3, 13, 'final', 0)

this one worked for me.

Django==2.0.0 Zappa==0.48.2

Python 3.6

Did you notice any performance tradeoffs between PyMySQL and mysqlclient?

[mathsmtnx]

Hey Guys. Is there some work around that can be done with this error using Flask Application? Im still facing here zappa=0.54.1 python=3.7