Sqlite3 database is locked error

[lovmo]

Describe the bug I receive a sqlite3.OperationalError: database is locked error when syncing with Stripe with webhook.

To Reproduce Steps to reproduce the behavior:

1. Enable stripe webhook to dj-stripe
2. Start local stripe webhook CLI
3. Run the command: stripe trigger checkout.session.completed
4. See attached error on webhook

Expected behavior I expect that django would handle the webhook without throwing a database is locked error. I want to use the webhook to update my account model.

If relevant it's very helpful to include webhook tracebacks and content (note that these are logged in the database and are visible in django admin - eg http://127.0.0.1:8000/admin/djstripe/webhookeventtrigger/ )

Traceback (most recent call last):
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 416, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:
Exception: database is locked

Traceback (most recent call last):
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/djstripe/models/webhooks.py", line 194, in from_request
obj.process(save=False)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/djstripe/models/webhooks.py", line 285, in process
self.event = Event.process(self.json_body)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/djstripe/models/core.py", line 1524, in process
ret = cls._create_from_stripe_object(data)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/djstripe/models/base.py", line 585, in _create_from_stripe_object
instance.save(force_insert=True)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/models/base.py", line 743, in save
self.save_base(using=using, force_insert=force_insert,
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/models/base.py", line 780, in save_base
updated = self._save_table(
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/models/base.py", line 885, in _save_table
results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/models/base.py", line 923, in _do_insert
return manager._insert(
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1301, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1441, in execute_sql
cursor.execute(sql, params)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 99, in execute
return super().execute(sql, params)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute
with self.db.wrap_database_errors:
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 416, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: database is locked

Environment

• dj-stripe version: 2.6.0
• Your Stripe account's default API version: 2020-08-27
• Database: sqlite3
• Python version: 3.10
• Django version: 4.0.1

Can you reproduce the issue with the latest version of master?

Yes

[jleclanche]

Will leave it as a valid issue but to be frank, sqlite backend is meant to be used at most for loose testing; anything production-worthy will be using a different backend. Webhooks not working on it is not completely unsurprising.

Still, will look into it.

[lovmo]

Thanks! Yeah, I will use Postgresql for production, but I have not bothered to set that up yet. The project is fairly new so using sqlite has been sufficient until now. Keep up the good work!

[Hafnernuss]

I can confirm this happening on SQLite, which is quite handy in development.

[fabge]

I can confirm this too. Although I would argue that sqlite can handle quite the load and a single webhook call should not lock the database.

[MicahLyle]

I think I've noticed that, even in PostgreSQL, since djstripe 2.6.0 (I could be wrong about the since which version), there seems to be more deadlocks on the Sentry logs.

Here's where it can become a problem. I have some older (like ~3 year old code) that does some sanity check things with default sources that looks like this.

    if set_default:
        stripe_customer.default_source = stripe_source["id"]
        stripe_customer.save()
    new_payment_method = DjstripePaymentMethod.from_stripe_object(stripe_source)
    if set_default:
        customer.default_source = new_payment_method
        customer.save()

That code is part of a critical production workflow that involves paying and activating something (all in one step). It's on the latest djstripe version.

For the first time ever, the other day, I had the production code there throw a 500 because of a deadlock with djstripe's webhook processing. My plan is to catch the deadlock exception by checking for that type of error with "deadlock detected" in the exception string, and then retry a number of times, maybe with time.sleep in-between since it's just so rare. I probably have to do it though because it's a critical endpoint that really shouldn't fail. I'm hopeful that will work fine, but, I'm not sure if I'll have to do extra work with the transaction due to a deadlock actually occurring where I'll have to abort the transaction and retry the entire request, etc.

[MicahLyle]

^ By the way, with the above comment, the resolution was to make sure to not try and create the customer and payment method together in the same request (we were only creating customers right about when they were adding payment info, because we were doing it an older way with sources using DjstripePaymentMethod. Creating the customer just earlier on and separately in the flow I believe resolves the deadlocks because the deadlock happens with the source/payment method created webhook and customer created webhook coming in concurrently, and source webhook wants to create customer and customer webhook wants to create default source and they deadlock (I think).

[jakemanger]

This appears to be very similar to my issue that only occurs when using djstripe 2.6. See #1625

[luzader]

A workaround is to disable threading in the development server using the command line switch:

manage.py runserver --nothreading

Even though stripe-cli can start multiple webhooks concurrently, this will ensure they are processed one-by-one, and thus no database-contention can occur. This workaround could hide concurrency problems in your project, but I'm not sure I could ever discover those through manual local testing anyway.

[Hafnernuss]

That sounds cool. Running a docker in development just to circumvent those issues is a pain anyway, so I will give this a try.

[jleclanche]

Closing as sqlite is not officially supported; i believe 3.0 will resolve this at any rate.