Snuba retention days configuration leads to a 'timedelta' error

Self-Hosted Version

22.8.0

CPU Architecture

x86_64

Docker Version

20.10.11

Docker Compose Version

2.9.0

Steps to Reproduce

Unknown, updated older install that had .env configured with:

SENTRY_EVENT_RETENTION_DAYS=45

Might've missed an upgrade step (e.g. sentry.conf.py?) as required changes across versions aren't clearly documented as far as I could tell.

See the comment on https://github.com/getsentry/snuba/issues/2780#issuecomment-1196314809

Expected Result

snuba-consumer and snuba-transactions-consumer to not crash.

~~Sadly, the fix in the comment is difficult to apply as the consumer container terminates instantly before I can even type apt insta to install nano.~~

Managed to make a custom snuba image with the fix pre-applied:

Dockerfile

FROM getsentry/snuba:22.8.0
RUN sed -i -e 's/# This is not ideal but it should never happen anyways/retention_days = int(retention_days)/g' /usr/src/snuba/snuba/datasets/events_format.py

Actual Result

snuba-consumer and snuba-transactions-consumer crash:

sentry_onpremise-snuba-transactions-consumer-1  | 2022-08-20 23:15:45,778 Processor terminated
sentry_onpremise-snuba-transactions-consumer-1  | Traceback (most recent call last):
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/src/snuba/snuba/consumers/consumer.py", line 656, in process_message
sentry_onpremise-snuba-transactions-consumer-1  |     result = processor.process_message(
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/src/snuba/snuba/datasets/transactions_processor.py", line 394, in process_message
sentry_onpremise-snuba-transactions-consumer-1  |     event_dict, retention_days = self._structure_and_validate_message(message) or (
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/src/snuba/snuba/datasets/transactions_processor.py", line 88, in _structure_and_validate_message
sentry_onpremise-snuba-transactions-consumer-1  |     retention_days = override_and_enforce_retention(
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/src/snuba/snuba/datasets/events_format.py", line 113, in override_and_enforce_retention
sentry_onpremise-snuba-transactions-consumer-1  |     return enforce_retention(retention_days, timestamp)
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/src/snuba/snuba/datasets/events_format.py", line 137, in enforce_retention
sentry_onpremise-snuba-transactions-consumer-1  |     datetime.utcnow() - timedelta(days=retention_days)
sentry_onpremise-snuba-transactions-consumer-1  | TypeError: unsupported type for timedelta days component: str
sentry_onpremise-snuba-transactions-consumer-1  |
sentry_onpremise-snuba-transactions-consumer-1  | The above exception was the direct cause of the following exception:
sentry_onpremise-snuba-transactions-consumer-1  |
sentry_onpremise-snuba-transactions-consumer-1  | Traceback (most recent call last):
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/local/bin/snuba", line 33, in <module>
sentry_onpremise-snuba-transactions-consumer-1  |     sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
sentry_onpremise-snuba-transactions-consumer-1  |     return self.main(*args, **kwargs)
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
sentry_onpremise-snuba-transactions-consumer-1  |     rv = self.invoke(ctx)
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
sentry_onpremise-snuba-transactions-consumer-1  |     return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
sentry_onpremise-snuba-transactions-consumer-1  |     return ctx.invoke(self.callback, **ctx.params)
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
sentry_onpremise-snuba-transactions-consumer-1  |     return __callback(*args, **kwargs)
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/src/snuba/snuba/cli/consumer.py", line 188, in consumer
sentry_onpremise-snuba-transactions-consumer-1  |     consumer.run()
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 131, in run
sentry_onpremise-snuba-transactions-consumer-1  |     self._run_once()
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 169, in _run_once
sentry_onpremise-snuba-transactions-consumer-1  |     self.__processing_strategy.submit(self.__message)
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/streaming/filter.py", line 32, in submit
sentry_onpremise-snuba-transactions-consumer-1  |     self.__next_step.submit(message)
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/streaming/transform.py", line 72, in submit
sentry_onpremise-snuba-transactions-consumer-1  |     self.__transform_function(message),
sentry_onpremise-snuba-transactions-consumer-1  |   File "/usr/src/snuba/snuba/consumers/consumer.py", line 664, in process_message
sentry_onpremise-snuba-transactions-consumer-1  |     raise InvalidMessages(

FTR this should have been fixed in https://github.com/getsentry/snuba/pull/2809

Did you follow the instructions at https://develop.sentry.dev/self-hosted/releases/#upgrading ? What version were you upgrading from?

If the install script didn't check for a watershed/hard-stop build in order to block the upgrade, it might be one or more of those were skipped over time, yes, as I wasn't aware of the existence of these steps. However, these all seem related to 2021 releases, so this may not apply.

The comment in the linked PR forcing another 'int' cast does seem to resolve this.

The installers do not enforce hard stops as far as I am aware, so missing hard stops is likely the issue.

The comment in the linked PR forcing another 'int' cast does seem to resolve this.

Glad to hear that at least, should we close this issue out then?

Note: 'the comment'. The PR itself does not include the same change from the comment, unless this was added in the Snuba repo since the latest tag at the time of posting this issue.

Similarly, removing the change leads to the processing failures returning.

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

Not stale.

Similarly, removing the change leads to the processing failures returning.

I think this change is normally handled as part of the upgrade process, so I don't believe it is needed otherwise.

Same issue here in the latest version. new events come in with null in retention days field.

Sorry, I somehow missed the comment in the snuba repo. I think this is follow up to that issue, so I am going to close this and leave a comment on that issue to see if more work is needed on the snuba side. I'm sorry I misunderstood the issue originally.