django-clickhouse-backend icon indicating copy to clipboard operation
django-clickhouse-backend copied to clipboard

[BUG ] UUID like id dont work afrter upate 1.0.1 -> 1.1.4

Open lakeofcolors opened this issue 2 years ago • 1 comments

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/./manage.py", line 22, in <module>
    main()
  File "/app/./manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/app/pkgs/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/app/pkgs/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/app/pkgs/django/core/management/base.py", line 402, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/app/pkgs/django/core/management/base.py", line 448, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/pkgs/django/core/management/base.py", line 96, in wrapped
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/pkgs/django/core/management/commands/migrate.py", line 376, in handle
    emit_post_migrate_signal(
  File "/app/pkgs/django/core/management/sql.py", line 52, in emit_post_migrate_signal
    models.signals.post_migrate.send(
  File "/app/pkgs/django/dispatch/dispatcher.py", line 176, in send
    return [
           ^
  File "/app/pkgs/django/dispatch/dispatcher.py", line 177, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/pkgs/django/contrib/auth/management/__init__.py", line 51, in create_permissions
    create_contenttypes(
  File "/app/pkgs/django/contrib/contenttypes/management/__init__.py", line 142, in create_contenttypes
    ContentType.objects.using(using).bulk_create(cts)
  File "/app/pkgs/django/db/models/query.py", line 816, in bulk_create
    returned_columns = self._batched_insert(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/app/pkgs/django/db/models/query.py", line 1825, in _batched_insert
    self._insert(
  File "/app/pkgs/django/db/models/query.py", line 1791, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/pkgs/clickhouse_backend/models/sql/compiler.py", line 150, in execute_sql
    cursor.execute(sql, params)
  File "/app/pkgs/django/db/backends/utils.py", line 102, in execute
    return super().execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/pkgs/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/pkgs/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/pkgs/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/app/pkgs/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/app/pkgs/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/pkgs/clickhouse_backend/driver/connection.py", line 105, in execute
    super().execute(operation, parameters)
  File "/app/pkgs/clickhouse_driver/dbapi/cursor.py", line 117, in execute
    raise OperationalError(orig)
django.db.utils.OperationalError: Code: 53. Type mismatch in VALUES section. Repeat query with types_check=True for detailed info. Column id: 'i' format requires -2147483648 <= number <= 2147483647

My base model

class UUIDBasedModel(models.ClickhouseModel):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)

    class Meta:
        abstract = True

Table in database is created but this exception

lakeofcolors avatar Nov 30 '23 15:11 lakeofcolors

I am sorry. I have made a breaking change from 1.0 to 1.1. SmallAutoField and AutoField used to be Int16 and Int32. From 1.1, they all become Int64, so that the default IdWorker can generate snowflake id for them.

All django contrib apps have models with AutoField as primary key. 1.0 and below, there is no auto primary generated for these models, their id fields will always be 0!

If you really want to use django contrib apps in your project, you can manully change id type. Here is an example:

show create table django_content_type;
create table django_content_type_new (id Int64, app_label FixedString(100), model FixedString(100)) Engine MergeTree order by id;
insert into django_content_type_new select * from django_content_type;
drop table django_content_type;
rename table django_content_type_new to django_content_type;

This issue is caused by post migration signal, django will create new contenttype.ContentType and auth.Permission for newly created models. When creating, idWorkd will generate Uint64 id for the records, while in the database id field have Int32 type.

  1. Because it's caused by post migration signal, indeed migrations have been applied already.
  2. contenttype.ContentType and auth.Permission of newly created models are not created. If they are important to you, you should create them yourself.

jayvynl avatar Dec 01 '23 14:12 jayvynl