bookwyrm
bookwyrm copied to clipboard
Published
20 hours ago •
bookwyrm-social
bookwyrm-social
bw-dev is probably not idempotent or fauilt-tolerant
Describe the bug
I most likely had a problem with S3 configuration when running ./bw-dev setup.
It ended with:
root@skobk:~/bookwyrm# ./bw-dev setup [119/848]
+ case "$CMD" in
+ migrate
+ runweb python manage.py migrate
+ docker compose run --rm web python manage.py migrate
[+] Creating 4/4
_ Container bookwyrm-db-1 Recreated 0.2s
_ Container bookwyrm-redis_activity-1 Recreated 0.2s
_ Container bookwyrm-redis_broker-1 Recreated 0.2s
_ Container bookwyrm-celery_worker-1 Recreated 0.1s
[+] Running 4/4
_ Container bookwyrm-db-1 Started 0.3s
_ Container bookwyrm-redis_broker-1 Started 0.6s
_ Container bookwyrm-redis_activity-1 Started 0.6s
_ Container bookwyrm-celery_worker-1 Started 0.3s
Operations to perform:
Apply all migrations: admin, auth, bookwyrm, contenttypes, django_celery_beat, sessions
Running migrations:
No migrations to apply.
+ migrate django_celery_beat
+ runweb python manage.py migrate django_celery_beat
+ docker compose run --rm web python manage.py migrate django_celery_beat
[+] Creating 4/0
_ Container bookwyrm-redis_broker-1 Running 0.0s
_ Container bookwyrm-db-1 Running 0.0s
_ Container bookwyrm-redis_activity-1 Running 0.0s
_ Container bookwyrm-celery_worker-1 Running 0.0s
Operations to perform:
Apply all migrations: django_celery_beat
Running migrations:
No migrations to apply.
+ initdb
+ runweb python manage.py initdb
+ docker compose run --rm web python manage.py initdb
[+] Creating 4/0
_ Container bookwyrm-redis_broker-1 Running 0.0s
_ Container bookwyrm-redis_activity-1 Running 0.0s
_ Container bookwyrm-db-1 Running 0.0s
_ Container bookwyrm-celery_worker-1 Running 0.0s
+ runweb python manage.py compile_themes
+ docker compose run --rm web python manage.py compile_themes
[+] Creating 4/0
_ Container bookwyrm-db-1 Running 0.0s
_ Container bookwyrm-redis_broker-1 Running 0.0s
_ Container bookwyrm-redis_activity-1 Running 0.0s
_ Container bookwyrm-celery_worker-1 Running 0.0s
Compiled SASS/SCSS file: '/app/bookwyrm/static/css/themes/bookwyrm-light.scss'
Compiled SASS/SCSS file: '/app/bookwyrm/static/css/themes/bookwyrm-dark.scss'
+ runweb python manage.py collectstatic --no-input
+ docker compose run --rm web python manage.py collectstatic --no-input
[+] Creating 4/0
_ Container bookwyrm-db-1 Running 0.0s
_ Container bookwyrm-redis_activity-1 Running 0.0s
_ Container bookwyrm-redis_broker-1 Running 0.0s
_ Container bookwyrm-celery_worker-1 Running 0.0s
Traceback (most recent call last):
File "/app/manage.py", line 18, in <module> [63/848]
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 187, in handle
collected = self.collect()
File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
handler(path, prefixed_path, storage)
File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 338, in copy_file
if not self.delete_file(path, prefixed_path, source_storage):
File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 248, in delete_file
if self.storage.exists(prefixed_path):
File "/usr/local/lib/python3.9/site-packages/storages/backends/s3boto3.py", line 463, in exists
self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 964, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request
I probably fixed the problem and to check that I ran setup again. But now setup can't set up the instance already because some keys were already persisted in the database:
root@skobk:~/bookwyrm# ./bw-dev setup [37/848]
+ case "$CMD" in
+ migrate
+ runweb python manage.py migrate
+ docker compose run --rm web python manage.py migrate
[+] Creating 4/4
_ Container bookwyrm-redis_broker-1 Recreated 0.8s
_ Container bookwyrm-db-1 Recreated 1.0s
_ Container bookwyrm-redis_activity-1 Recreated 0.6s
_ Container bookwyrm-celery_worker-1 Recreated 2.1s
[+] Running 4/4
_ Container bookwyrm-redis_broker-1 Started 0.6s
_ Container bookwyrm-db-1 Started 0.5s
_ Container bookwyrm-redis_activity-1 Started 0.5s
_ Container bookwyrm-celery_worker-1 Started 0.3s
Operations to perform:
Apply all migrations: admin, auth, bookwyrm, contenttypes, django_celery_beat, sessions
Running migrations:
No migrations to apply.
+ migrate django_celery_beat
+ runweb python manage.py migrate django_celery_beat
+ docker compose run --rm web python manage.py migrate django_celery_beat
[+] Creating 4/0
_ Container bookwyrm-db-1 Running 0.0s
_ Container bookwyrm-redis_broker-1 Running 0.0s
_ Container bookwyrm-redis_activity-1 Running 0.0s
_ Container bookwyrm-celery_worker-1 Running 0.0s
Operations to perform:
Apply all migrations: django_celery_beat
Running migrations:
No migrations to apply.
+ initdb
+ runweb python manage.py initdb
+ docker compose run --rm web python manage.py initdb
[+] Creating 4/0
_ Container bookwyrm-redis_broker-1 Running 0.0s
_ Container bookwyrm-redis_activity-1 Running 0.0s
_ Container bookwyrm-db-1 Running 0.0s
_ Container bookwyrm-celery_worker-1 Running 0.0s
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "bookwyrm_connector_identifier_key"
DETAIL: Key (identifier)=(bookwyrm.social) already exists.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/manage.py", line 18, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/app/bookwyrm/management/commands/initdb.py", line 179, in handle
init_connectors()
File "/app/bookwyrm/management/commands/initdb.py", line 80, in init_connectors
models.Connector.objects.create(
File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 453, in create
obj.save(force_insert=True, using=self.db)
File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 739, in save
self.save_base(using=using, force_insert=force_insert,
File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
updated = self._save_table(
File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table
results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert
return manager._insert(
File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "bookwyrm_connector_identifier_key"
DETAIL: Key (identifier)=(bookwyrm.social) already exists.
Expected behavior Properly processing setup procedure if it is running again after failure
Instance Is not online yet.
You're completely right that it should be possible to re-run this command safely! We should fix that.
As an intermediary fix just to get you moving for now, it looks like the database setup part worked correctly and the command stopped at collectstatic, so you should be able to address the issue with collectstatic (it sounds like it might have been an s3 config thing?) and then just run
./bw-dev collectstatic
./bw-dev admin_code
And that should get you the rest of the way through the command
Sorry, I wasn't able to check if your fix works out. As I didn't have anything that should be preserved in the database yet I just pruned Docker volumes and started again without S3 to try to move to it later when I have more time. It worked as expected.
But thank you for a quick reply!