self-hosted
self-hosted copied to clipboard
getsentry
restarting post-process-forwarder-transactions and post-process-forwarder-errors
Self-Hosted Version
23.6.1
CPU Architecture
x86_64
Docker Version
20.10.16
Docker Compose Version
2.6.0
Steps to Reproduce
post-process-forwarder-errors container docker logs shows: Error: No such option: --synchronize-commit-log-topic
command in doclker-compose.yml command: run consumer post-process-forwarder-errors --consumer-group post-process-forwarder --synchronize-commit-log-topic=snuba-commit-log --synchronize-commit-group=snuba-consumers
AND
post-process-forwarder-transactions docker-logs shows: Error: No such option: --synchronize-commit-log-topic
command in doclker-compose.yml command: command: run consumer post-process-forwarder-transactions --consumer-group post-process-forwarder --synchronize-commit-log-topic=snuba-transactions-commit-log --synchronize-commit-group transactions_group
Expected Result
these containers restarting
Actual Result
docker logs post-process-forwarder-transactions Usage: sentry run consumer [OPTIONS] CONSUMER_NAME [CONSUMER_ARGS]... Try 'sentry run consumer --help' for help.
Error: No such option: --synchronize-commit-log-topic Usage: sentry run consumer [OPTIONS] CONSUMER_NAME [CONSUMER_ARGS]... Try 'sentry run consumer --help' for help.
Error: No such option: --synchronize-commit-log-topic Usage: sentry run consumer [OPTIONS] CONSUMER_NAME [CONSUMER_ARGS]... Try 'sentry run consumer --help' for help.
docker logs post-process-forwarder-errors Usage: sentry run consumer [OPTIONS] CONSUMER_NAME [CONSUMER_ARGS]... Try 'sentry run consumer --help' for help.
Error: No such option: --synchronize-commit-log-topic Usage: sentry run consumer [OPTIONS] CONSUMER_NAME [CONSUMER_ARGS]... Try 'sentry run consumer --help' for help.
Error: No such option: --synchronize-commit-log-topic Usage: sentry run consumer [OPTIONS] CONSUMER_NAME [CONSUMER_ARGS]... Try 'sentry run consumer --help' for help.
Event ID
No response
Interesting, are you by chance using 23.6.1 images but have a docker-compose file that is up to date with the master branch? the synchronize-commit-log-topic option wasn't added in self-hosted until last week.
https://github.com/getsentry/self-hosted/pull/2224
Yes just today downloaded master branch, well restored commnads from 23.6.0 not restarting any more.... But question is is working everything fine? who can investigate and fix ?
If you downloaded the master branch, you will need to use nightly images for your self-hosted instance to work instead of 23.6.1. Everything as far as I can tell is working as intended
hmm nightly not same 23.6.1 for today? maybe you can tell build/tag name i prefear to lock image versions
Nightly is not the same as 23.6.1. You'll need to restore your self-hosted instance to use 23.6.1 instead of using master to resolve the issue here.
ok thanks i will try tomorrow
On 2023-06-28, Wed at 20:22, Hubert Deng @.***> wrote:
Nightly is not the same as 23.6.1. You'll need to restore your self-hosted instance to use 23.6.1 instead of using master to resolve the issue here.
— Reply to this email directly, view it on GitHub https://github.com/getsentry/self-hosted/issues/2242#issuecomment-1611807776, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRXGSOABUPVYVCU35RBGXDXNRR5PANCNFSM6AAAAAAZXDBDCY . You are receiving this because you authored the thread.Message ID: @.***>
works but Sentry 23.7.0.dev0 doesn't look like stable.... can you please tell me 23.6.1 working build? NOT dev
works but Sentry 23.7.0.dev0 doesn't look like stable.... can you please tell me 23.6.1 working build? NOT dev
Sorry, what do you mean when you say this? 23.6.1 should be stable for you
Any Updates? I'm having the same Issues with 24.2.0...
08:47:00 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='transactions'), index=0): 109430346}
08:47:00 [INFO] sentry.post_process_forwarder.post_process_forwarder: Starting multithreaded post process forwarder
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/arroyo/processing/processor.py", line 319, in run
self._run_once()
File "/usr/local/lib/python3.11/site-packages/arroyo/processing/processor.py", line 381, in _run_once
self.__message = self.__consumer.poll(timeout=1.0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sentry/consumers/synchronized.py", line 235, in poll
message = self.__consumer.poll(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/arroyo/backends/kafka/consumer.py", line 414, in poll
raise OffsetOutOfRange(str(error))
arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"}
08:47:00 [ERROR] arroyo.processing.processor: Caught exception, shutting down...
08:47:00 [INFO] arroyo.processing.processor: Closing <sentry.consumers.synchronized.SynchronizedConsumer object at 0x7f3160adec10>...
08:47:00 [INFO] arroyo.processing.processor: Partitions to revoke: [Partition(topic=Topic(name='transactions'), index=0)]
08:47:00 [INFO] arroyo.processing.processor: Partition revocation complete.
08:47:00 [INFO] arroyo.processing.processor: Processor terminated
Traceback (most recent call last):
File "/usr/local/bin/sentry", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.11/site-packages/sentry/runner/__init__.py", line 190, in main
func(**kwargs)
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sentry/runner/decorators.py", line 69, in inner
return ctx.invoke(f, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sentry/runner/decorators.py", line 29, in inner
return ctx.invoke(f, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sentry/runner/commands/run.py", line 557, in basic_consumer
run_processor_with_signals(processor, consumer_name)
File "/usr/local/lib/python3.11/site-packages/sentry/utils/kafka.py", line 46, in run_processor_with_signals
processor.run()
File "/usr/local/lib/python3.11/site-packages/arroyo/processing/processor.py", line 319, in run
self._run_once()
File "/usr/local/lib/python3.11/site-packages/arroyo/processing/processor.py", line 381, in _run_once
self.__message = self.__consumer.poll(timeout=1.0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sentry/consumers/synchronized.py", line 235, in poll
message = self.__consumer.poll(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/arroyo/backends/kafka/consumer.py", line 414, in poll
raise OffsetOutOfRange(str(error))
arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"}
Have you tried this approach?
But to answer your question, we have not looked at this issue specifically.
This issue has gone three weeks without activity. In another week, I will close it.
But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!
"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀
![getsantry[bot] avatar](https://avatars.githubusercontent.com/in/66573?v=4 "Published by a pub.dev verified publisher"){kind=small}