Backup is broken: column sentry_customdynamicsamplingrule.query does not exist

[ingria]

Self-Hosted Version

23.10.1 (master branch)

CPU Architecture

x86_x64

Docker Version

24.0.4

Docker Compose Version

2.18.1

Steps to Reproduce

./scripts/backup.sh

Expected Result

valid backup.json file

Actual Result

Container sentry-self-hosted-snuba-replacer-1  Started
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
/usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if key is '':
/usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if key_extra_len is 0:
RpcExportError(
    kind: Unknown,
    on: InstanceID(model: 'sentry.customdynamicsamplingrule'),
    reason: Unknown internal error occurred: UndefinedColumn('column sentry_customdynamicsamplingrule.query does not exist\nLINE 1: ...ntry_customdynamicsamplingrule"."condition_hash", "sentry_cu...\n                                                             ^\n')
SQL: SELECT "sentry_customdynamicsamplingrule"."id", "sentry_customdynamicsamplingrule"."date_added", "sentry_customdynamicsamplingrule"."organization_id", "sentry_customdynamicsamplingrule"."is_active", "sentry_customdynamicsamplingrule"."is_org_level", "sentry_customdynamicsamplingrule"."rule_id", "sentry_customdynamicsamplingrule"."condition", "sentry_customdynamicsamplingrule"."sample_rate", "sentry_customdynamicsamplingrule"."start_date", "sentry_customdynamicsamplingrule"."end_date", "sentry_customdynamicsamplingrule"."num_samples", "sentry_customdynamicsamplingrule"."condition_hash", "sentry_customdynamicsamplingrule"."query", "sentry_customdynamicsamplingrule"."created_by_id" FROM "sentry_customdynamicsamplingrule" WHERE "sentry_customdynamicsamplingrule"."id" > %s ORDER BY "sentry_customdynamicsamplingrule"."id" ASC
)
RpcExportError(
    kind: Unknown,
    on: InstanceID(model: 'sentry.customdynamicsamplingrule'),
    reason: Unknown internal error occurred: UndefinedColumn('column sentry_customdynamicsamplingrule.query does not exist\nLINE 1: ...ntry_customdynamicsamplingrule"."condition_hash", "sentry_cu...\n                                                             ^\n')
SQL: SELECT "sentry_customdynamicsamplingrule"."id", "sentry_customdynamicsamplingrule"."date_added", "sentry_customdynamicsamplingrule"."organization_id", "sentry_customdynamicsamplingrule"."is_active", "sentry_customdynamicsamplingrule"."is_org_level", "sentry_customdynamicsamplingrule"."rule_id", "sentry_customdynamicsamplingrule"."condition", "sentry_customdynamicsamplingrule"."sample_rate", "sentry_customdynamicsamplingrule"."start_date", "sentry_customdynamicsamplingrule"."end_date", "sentry_customdynamicsamplingrule"."num_samples", "sentry_customdynamicsamplingrule"."condition_hash", "sentry_customdynamicsamplingrule"."query", "sentry_customdynamicsamplingrule"."created_by_id" FROM "sentry_customdynamicsamplingrule" WHERE "sentry_customdynamicsamplingrule"."id" > %s ORDER BY "sentry_customdynamicsamplingrule"."id" ASC
)
Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 195, in main
    func(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 671, in export_global
    raise e
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 662, in export_global
    export_in_global_scope(
  File "/usr/local/lib/python3.8/site-packages/sentry/backup/exports.py", line 235, in export_in_global_scope
    return _export(
  File "/usr/local/lib/python3.8/site-packages/sentry/backup/exports.py", line 125, in _export
    raise ExportingError(result)
sentry.backup.exports.ExportingError: is_err=True kind='Unknown' on=InstanceID(model='sentry.customdynamicsamplingrule', ordinal=None) left_pk=None right_pk=None reason='Unknown internal error occurred: UndefinedColumn(\'column sentry_customdynamicsamplingrule.query does not exist\\nLINE 1: ...ntry_customdynamicsamplingrule"."condition_hash", "sentry_cu...\\n                                                             ^\\n\')\nSQL: SELECT "sentry_customdynamicsamplingrule"."id", "sentry_customdynamicsamplingrule"."date_added", "sentry_customdynamicsamplingrule"."organization_id", "sentry_customdynamicsamplingrule"."is_active", "sentry_customdynamicsamplingrule"."is_org_level", "sentry_customdynamicsamplingrule"."rule_id", "sentry_customdynamicsamplingrule"."condition", "sentry_customdynamicsamplingrule"."sample_rate", "sentry_customdynamicsamplingrule"."start_date", "sentry_customdynamicsamplingrule"."end_date", "sentry_customdynamicsamplingrule"."num_samples", "sentry_customdynamicsamplingrule"."condition_hash", "sentry_customdynamicsamplingrule"."query", "sentry_customdynamicsamplingrule"."created_by_id" FROM "sentry_customdynamicsamplingrule" WHERE "sentry_customdynamicsamplingrule"."id" > %s ORDER BY "sentry_customdynamicsamplingrule"."id" ASC'
Error in scripts/_lib.sh:65.
'$dc run -v $(pwd)/sentry:/sentry-data/backup --rm -T -e SENTRY_LOG_LEVEL=CRITICAL web export global /sentry-data/backup/backup.json' exited with status 1
-> ./scripts/backup.sh:main:4
--> scripts/_lib.sh:backup:65

Event ID

No response

[azaslavsky]

What version of Sentry did you use to generate the backup? And what version are you importing on to? Are they both 23.10.1?

[ingria]

I've updated to the latest master (web ui shows Sentry 23.11.0.dev 0ea10232) and now I don't get this error (I do get another error though).

[azaslavsky]

What error do you get?

The original error you saw is almost certainly related to this migration not being applied: https://github.com/getsentry/sentry/blob/e0a04bbbf7f0ff90b43c66b910963d4340c88542/src/sentry/migrations/0578_add_query_and_users_to_custom_dynamic_sampling_rules.py

Are you sure you ran the upgrade script when pulling the new version? If the migrations all run, and you still see the error, please post it here.