lago
lago copied to clipboard
[BUG]: docker containers don't crash
Describe the bug If the database, or redis, is unavailable, the docker containers don't crash. This stops them from auto-healing (e.g. DNS recovering and injecting the env var).
To Reproduce Steps to reproduce the behavior:
- Deploy e.g. api-worker, the database, but leave out the redis URI
- The api-worker will now crash-loop internally and log a lot of output (= $$$) without actually crashing the Kubernetes pod.
api-worker-79b46fdf88-7nrnw api-worker W, [2023-02-10T13:33:44.322529 #7] WARN -- : /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:398:in `rescue in establish_connection'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:379:in `establish_connection'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:115:in `block in connect'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:344:in `with_reconnect'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:114:in `connect'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:25:in `block in connect'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:10:in `block in connect_with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:55:in `block in with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/tracer.rb:356:in `capture_segment_error'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:55:in `with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:10:in `connect_with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:25:in `connect'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:417:in `ensure_connected'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:269:in `block in process'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:356:in `logging'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sentry-ruby-core-5.3.1/lib/sentry/redis.rb:78:in `block in logging'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sentry-ruby-core-5.3.1/lib/sentry/redis.rb:17:in `block in instrument'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sentry-ruby-core-5.3.1/lib/sentry/redis.rb:28:in `record_span'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sentry-ruby-core-5.3.1/lib/sentry/redis.rb:16:in `instrument'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sentry-ruby-core-5.3.1/lib/sentry/redis.rb:77:in `logging'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:268:in `process'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:161:in `call'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:17:in `block in call'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:17:in `block in call_with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:55:in `block in with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/tracer.rb:356:in `capture_segment_error'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:55:in `with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:17:in `call_with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:17:in `call'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis.rb:269:in `block in send_command'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis.rb:268:in `synchronize'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis.rb:268:in `send_command'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/commands/sets.rb:11:in `scard'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/api.rb:867:in `block in size'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq.rb:156:in `block in redis'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/connection_pool-2.3.0/lib/connection_pool.rb:65:in `block (2 levels) in with'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/connection_pool-2.3.0/lib/connection_pool.rb:64:in `handle_interrupt'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/connection_pool-2.3.0/lib/connection_pool.rb:64:in `block in with'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/connection_pool-2.3.0/lib/connection_pool.rb:61:in `handle_interrupt'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/connection_pool-2.3.0/lib/connection_pool.rb:61:in `with'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq.rb:153:in `redis'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/api.rb:867:in `size'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb:190:in `process_count'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb:151:in `random_poll_interval'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb:120:in `wait'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb:102:in `block in start'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/component.rb:8:in `watchdog'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/component.rb:17:in `block in safe_thread'
Expected behavior If the environment a software service runs in is incorrectly configured, print an error message and exit the process. This lets monitoring software alert to the problem. Otherwise ops has to write app-specific event listeners, eventify the logs (including writing stacktrace parsers for ruby) and deploy listeners for the logs, then write API-integrations with the platform runtime (like kubernetes) to restart the pod.
Support
- Version: getlago/api:v0.22.0-beta
Hi @haf - thank you for your message!
It's definitely something that should be improved. We will investigate and get back to you with a fix as soon as possible.
Have we any status on this? Seem to be running into something sort of related where the API suddendly returns 500, and a compose down/up fixes it
@OscarKolsrud we have to dig it a bit since this is the way Rails/Sidekiq works today, we may have to make it customizable. On our side, on production, we prefer to have errors instead of pods always restarting, but it's a Rubyist habit I think 😂
Hi, my org also noticed this problem, we ended up having a lot of problems because of the lago clock being frozen on error state for 2 days after a redis disconnect (we noticed that it just stopped logging after the error, although running, but customers weren't charged). It's the second or third time we lose a bill-customers day (we suspect) because of this type of outage (pods in error state in last/first days of months). Do you have updates on this, or any suggestion of a mechanism to restart pods once they enter error state?
https://en.wikipedia.org/wiki/Crash-only_software - this is a reference to a very sane way of building software - where you auto-correct faults by restarting at different levels.
@gabrielseibel1 if Redis is your problem, restarting the pod will not fix the issue, it will restart since Redis will be available again. What are the errors you face on the worker when it's in error?
If the connectivity to redis is a problem, a restart will trigger a retry of the connection
each time a job is enqueued or want to run, the connection to redis is retried. Is the health check enough for you guys? If the database or redis is out, health endpoint will return an error
If it's done using a liveness probe in k8s it would have solved my problem. If you've solved the bug from the stacktrace in the original post in this thread, you can close the issue. Do note that this stacktrace happens on container start though; not after a while / for networking errors - so in the case of this issue - you'd never have a successful health check.
I'm currently working on the liveness probe on our helmchart so this is definitely something we'll release very soon!
it's been a year. the clock crashes often. how do you guys deal with this?
@doctorpangloss we never had any clock crash for our cloud environment. If it crash, it's always because of an unhealthy Redis service.
Here is another log from broken clock with lago v1.4. The pod of the lago-clock did not recover within 2 days. There are no errors in the redis logs (newest log from redis is 2024-07-12T05:23:25.509128941Z)
2024-07-13T03:07:09.001473987Z I, [2024-07-13T03:07:09.001191 #7] INFO -- : Triggering 'schedule:activate_subscriptions'
2024-07-13T03:07:09.004434441Z I, [2024-07-13T03:07:09.004332 #7] INFO -- : [ActiveJob] Enqueued Clock::ActivateSubscriptionsJob (Job ID: 747ec5ec-3867-4952-bb75-8d954ddd3d3e) to Sidekiq(clock)
2024-07-13T03:07:09.004451474Z I, [2024-07-13T03:07:09.004374 #7] INFO -- : Finished 'schedule:activate_subscriptions' duration_ms=3 error=nil
2024-07-13T03:07:09.004459389Z I, [2024-07-13T03:07:09.004417 #7] INFO -- : Triggering 'schedule:refresh_draft_invoices'
2024-07-13T03:07:09.005629697Z I, [2024-07-13T03:07:09.005529 #7] INFO -- : [ActiveJob] Enqueued Clock::RefreshDraftInvoicesJob (Job ID: e5080ed5-e1df-49cd-a00d-6dbed2cafd4d) to Sidekiq(clock)
2024-07-13T03:07:09.005680141Z I, [2024-07-13T03:07:09.005584 #7] INFO -- : Finished 'schedule:refresh_draft_invoices' duration_ms=1 error=nil
2024-07-13T03:10:00.002022998Z I, [2024-07-13T03:10:00.001190 #7] INFO -- : Triggering 'schedule:bill_customers'
2024-07-13T03:10:00.003415444Z I, [2024-07-13T03:10:00.003268 #7] INFO -- : [ActiveJob] Enqueued Clock::SubscriptionsBillerJob (Job ID: 34ca702a-2be8-4fc5-9775-99396cb619d5) to Sidekiq(clock)
2024-07-13T03:10:00.003464144Z I, [2024-07-13T03:10:00.003396 #7] INFO -- : Finished 'schedule:bill_customers' duration_ms=2 error=nil
2024-07-13T03:12:09.001506411Z I, [2024-07-13T03:12:09.001196 #7] INFO -- : Triggering 'schedule:activate_subscriptions'
2024-07-13T03:12:09.211433834Z E, [2024-07-13T03:12:09.211235 #7] ERROR -- : [ActiveJob] Failed enqueuing Clock::ActivateSubscriptionsJob to Sidekiq(clock): Redis::TimeoutError (Waited 0.1 seconds)
2024-07-13T03:12:09.221068222Z E, [2024-07-13T03:12:09.219873 #7] ERROR -- : Waited 0.1 seconds (Redis::TimeoutError)
2024-07-13T03:12:09.221098369Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:140:in `block in fill_buffer'
2024-07-13T03:12:09.221104220Z <internal:kernel>:187:in `loop'
2024-07-13T03:12:09.221108678Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:122:in `fill_buffer'
2024-07-13T03:12:09.221112105Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:114:in `ensure_remaining'
2024-07-13T03:12:09.221115431Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:85:in `getbyte'
2024-07-13T03:12:09.221119058Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/resp3.rb:113:in `parse'
2024-07-13T03:12:09.221122334Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/resp3.rb:50:in `load'
2024-07-13T03:12:09.221126482Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection.rb:94:in `read'
2024-07-13T03:12:09.221130499Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/connection_mixin.rb:31:in `call'
2024-07-13T03:12:09.221134126Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:239:in `block (2 levels) in call_v'
2024-07-13T03:12:09.221137502Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/middlewares.rb:16:in `call'
2024-07-13T03:12:09.221159263Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:238:in `block in call_v'
2024-07-13T03:12:09.221163120Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:643:in `ensure_connected'
2024-07-13T03:12:09.221166937Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:237:in `call_v'
2024-07-13T03:12:09.221171877Z /usr/local/bundle/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
2024-07-13T03:12:09.221175864Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:12:in `block in call_v'
2024-07-13T03:12:09.221183719Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:19:in `block in call_with_tracing'
2024-07-13T03:12:09.221187226Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:59:in `block in with_tracing'
2024-07-13T03:12:09.221190581Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
2024-07-13T03:12:09.221194008Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:59:in `with_tracing'
2024-07-13T03:12:09.221200971Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:19:in `call_with_tracing'
2024-07-13T03:12:09.221204458Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:12:in `call_v'
2024-07-13T03:12:09.221207844Z /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
2024-07-13T03:12:09.221211201Z /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
2024-07-13T03:12:09.221214607Z /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
2024-07-13T03:12:09.221222892Z /usr/local/bundle/gems/redis-5.0.6/lib/redis/commands/scripting.rb:110:in `_eval'
2024-07-13T03:12:09.221226549Z /usr/local/bundle/gems/redis-5.0.6/lib/redis/commands/scripting.rb:97:in `evalsha'
2024-07-13T03:12:09.221229875Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:171:in `block (2 levels) in lock'
2024-07-13T03:12:09.221262186Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:152:in `with'
2024-07-13T03:12:09.221269359Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:171:in `block in lock'
2024-07-13T03:12:09.221273137Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:208:in `recover_from_script_flush'
2024-07-13T03:12:09.221276623Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:170:in `lock'
2024-07-13T03:12:09.221280339Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `block (2 levels) in lock_instances'
2024-07-13T03:12:09.221283746Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `select'
2024-07-13T03:12:09.221287192Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `block in lock_instances'
2024-07-13T03:12:09.221290919Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:311:in `timed'
2024-07-13T03:12:09.221312510Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:257:in `lock_instances'
2024-07-13T03:12:09.221316728Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:232:in `block in try_lock_instances'
2024-07-13T03:12:09.221320385Z <internal:numeric>:237:in `times'
2024-07-13T03:12:09.221323831Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:228:in `try_lock_instances'
2024-07-13T03:12:09.221327969Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:69:in `lock'
2024-07-13T03:12:09.221331786Z /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/strategies/base.rb:24:in `lock'
2024-07-13T03:12:09.221335433Z /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/strategies/base.rb:59:in `before_enqueue'
2024-07-13T03:12:09.221357564Z /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/active_job_patch.rb:50:in `block (2 levels) in <module:ActiveJobPatch>'
2024-07-13T03:12:09.221362213Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:467:in `instance_exec'
2024-07-13T03:12:09.221365920Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:467:in `block in make_lambda'
2024-07-13T03:12:09.221369777Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:199:in `block (2 levels) in halting'
2024-07-13T03:12:09.221373534Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:687:in `block (2 levels) in default_terminator'
2024-07-13T03:12:09.221377231Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:686:in `catch'
2024-07-13T03:12:09.221380718Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:686:in `block in default_terminator'
2024-07-13T03:12:09.221384204Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:200:in `block in halting'
2024-07-13T03:12:09.221388052Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `block in invoke_before'
2024-07-13T03:12:09.221394344Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `each'
2024-07-13T03:12:09.221397839Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `invoke_before'
2024-07-13T03:12:09.221401306Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:116:in `block in run_callbacks'
2024-07-13T03:12:09.221404893Z /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:25:in `block in instrument'
2024-07-13T03:12:09.221408339Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications.rb:206:in `block in instrument'
2024-07-13T03:12:09.221411786Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
2024-07-13T03:12:09.221415142Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications.rb:206:in `instrument'
2024-07-13T03:12:09.221418579Z /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:35:in `instrument'
2024-07-13T03:12:09.221422015Z /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:9:in `block (2 levels) in <module:Instrumentation>'
2024-07-13T03:12:09.221425442Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
2024-07-13T03:12:09.221428887Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
2024-07-13T03:12:09.221432735Z /usr/local/bundle/gems/activejob-traceable-0.4.2/lib/activejob/traceable/logging_patch.rb:17:in `block in tag_logger'
2024-07-13T03:12:09.221436532Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:99:in `block in tagged'
2024-07-13T03:12:09.221465366Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:37:in `tagged'
2024-07-13T03:12:09.221474193Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:99:in `tagged'
2024-07-13T03:12:09.221477940Z /usr/local/bundle/gems/activejob-traceable-0.4.2/lib/activejob/traceable/logging_patch.rb:17:in `tag_logger'
2024-07-13T03:12:09.221481667Z /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/logging.rb:14:in `block (2 levels) in <module:Logging>'
2024-07-13T03:12:09.221485604Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
2024-07-13T03:12:09.221523044Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
2024-07-13T03:12:09.221528594Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:138:in `run_callbacks'
2024-07-13T03:12:09.221532362Z /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/enqueuing.rb:63:in `enqueue'
2024-07-13T03:12:09.221564893Z #<Thread:0x00007f93a8327bc8 /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:426 run> terminated with exception (report_on_exception is true):
2024-07-13T03:12:09.221567518Z /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/enqueuing.rb:30:in `perform_later'
2024-07-13T03:12:09.221601682Z /app/clock.rb:22:in `block in <module:Clockwork>'
2024-07-13T03:12:09.221608213Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:61:in `execute'
2024-07-13T03:12:09.221613473Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:48:in `run'
2024-07-13T03:12:09.221618873Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:136:in `block in tick'
2024-07-13T03:12:09.221624624Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `each'
2024-07-13T03:12:09.221633461Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `tick'
2024-07-13T03:12:09.221639402Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:123:in `block (2 levels) in run_tick_loop'
2024-07-13T03:12:09.221647257Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `synchronize'
2024-07-13T03:12:09.221651615Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `block in run_tick_loop'
2024-07-13T03:12:09.221674437Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:435:in `block (2 levels) in thread_block_with_current_transaction'
2024-07-13T03:12:09.221686410Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
2024-07-13T03:12:09.221692241Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:434:in `block in thread_block_with_current_transaction'
2024-07-13T03:12:09.221699855Z I, [2024-07-13T03:12:09.221140 #7] INFO -- : Finished 'schedule:activate_subscriptions' duration_ms=220 error=#<Redis::TimeoutError: Waited 0.1 seconds>
2024-07-13T03:12:09.230473462Z /app/clock.rb:13:in `block in <module:Clockwork>': undefined local variable or method `e' for module Clockwork (NameError)
2024-07-13T03:12:09.230529367Z
2024-07-13T03:12:09.230555395Z Rails.logger.error(e.message)
2024-07-13T03:12:09.230576966Z ^
2024-07-13T03:12:09.230623443Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:154:in `handle_error'
2024-07-13T03:12:09.230655022Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:65:in `rescue in execute'
2024-07-13T03:12:09.230688184Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:62:in `execute'
2024-07-13T03:12:09.230748918Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:48:in `run'
2024-07-13T03:12:09.230812547Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:136:in `block in tick'
2024-07-13T03:12:09.230855647Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `each'
2024-07-13T03:12:09.230886325Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `tick'
2024-07-13T03:12:09.230943452Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:123:in `block (2 levels) in run_tick_loop'
2024-07-13T03:12:09.230984449Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `synchronize'
2024-07-13T03:12:09.231049311Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `block in run_tick_loop'
2024-07-13T03:12:09.231117679Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:435:in `block (2 levels) in thread_block_with_current_transaction'
2024-07-13T03:12:09.231194723Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
2024-07-13T03:12:09.231231221Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:434:in `block in thread_block_with_current_transaction'
2024-07-13T03:12:09.231268721Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:140:in `block in fill_buffer': Waited 0.1 seconds (Redis::TimeoutError)
2024-07-13T03:12:09.231363408Z from <internal:kernel>:187:in `loop'
2024-07-13T03:12:09.231403794Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:122:in `fill_buffer'
2024-07-13T03:12:09.231436616Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:114:in `ensure_remaining'
2024-07-13T03:12:09.231491719Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:85:in `getbyte'
2024-07-13T03:12:09.231538185Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/resp3.rb:113:in `parse'
2024-07-13T03:12:09.231592478Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/resp3.rb:50:in `load'
2024-07-13T03:12:09.231656918Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection.rb:94:in `read'
2024-07-13T03:12:09.231720717Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/connection_mixin.rb:31:in `call'
2024-07-13T03:12:09.231763988Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:239:in `block (2 levels) in call_v'
2024-07-13T03:12:09.231807509Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/middlewares.rb:16:in `call'
2024-07-13T03:12:09.231861712Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:238:in `block in call_v'
2024-07-13T03:12:09.231958883Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:643:in `ensure_connected'
2024-07-13T03:12:09.231979983Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:237:in `call_v'
2024-07-13T03:12:09.231986115Z from /usr/local/bundle/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
2024-07-13T03:12:09.231989280Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:12:in `block in call_v'
2024-07-13T03:12:09.231992356Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:19:in `block in call_with_tracing'
2024-07-13T03:12:09.231995041Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:59:in `block in with_tracing'
2024-07-13T03:12:09.231998257Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
2024-07-13T03:12:09.232000832Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:59:in `with_tracing'
2024-07-13T03:12:09.232004729Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:19:in `call_with_tracing'
2024-07-13T03:12:09.232007715Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:12:in `call_v'
2024-07-13T03:12:09.232010470Z from /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
2024-07-13T03:12:09.232013055Z from /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
2024-07-13T03:12:09.232032120Z from /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
2024-07-13T03:12:09.232035176Z from /usr/local/bundle/gems/redis-5.0.6/lib/redis/commands/scripting.rb:110:in `_eval'
2024-07-13T03:12:09.232038052Z from /usr/local/bundle/gems/redis-5.0.6/lib/redis/commands/scripting.rb:97:in `evalsha'
2024-07-13T03:12:09.232040647Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:171:in `block (2 levels) in lock'
2024-07-13T03:12:09.232043322Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:152:in `with'
2024-07-13T03:12:09.232045847Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:171:in `block in lock'
2024-07-13T03:12:09.232049412Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:208:in `recover_from_script_flush'
2024-07-13T03:12:09.232052889Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:170:in `lock'
2024-07-13T03:12:09.232055464Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `block (2 levels) in lock_instances'
2024-07-13T03:12:09.232058019Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `select'
2024-07-13T03:12:09.232108453Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `block in lock_instances'
2024-07-13T03:12:09.232126668Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:311:in `timed'
2024-07-13T03:12:09.232131767Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:257:in `lock_instances'
2024-07-13T03:12:09.232135524Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:232:in `block in try_lock_instances'
2024-07-13T03:12:09.232139691Z from <internal:numeric>:237:in `times'
2024-07-13T03:12:09.232143198Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:228:in `try_lock_instances'
2024-07-13T03:12:09.232146534Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:69:in `lock'
2024-07-13T03:12:09.232150111Z from /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/strategies/base.rb:24:in `lock'
2024-07-13T03:12:09.232153758Z from /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/strategies/base.rb:59:in `before_enqueue'
2024-07-13T03:12:09.232158868Z from /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/active_job_patch.rb:50:in `block (2 levels) in <module:ActiveJobPatch>'
2024-07-13T03:12:09.232162405Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:467:in `instance_exec'
2024-07-13T03:12:09.232166222Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:467:in `block in make_lambda'
2024-07-13T03:12:09.232169638Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:199:in `block (2 levels) in halting'
2024-07-13T03:12:09.232173284Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:687:in `block (2 levels) in default_terminator'
2024-07-13T03:12:09.232176951Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:686:in `catch'
2024-07-13T03:12:09.232180558Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:686:in `block in default_terminator'
2024-07-13T03:12:09.232184175Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:200:in `block in halting'
2024-07-13T03:12:09.232187652Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `block in invoke_before'
2024-07-13T03:12:09.232191068Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `each'
2024-07-13T03:12:09.232194545Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `invoke_before'
2024-07-13T03:12:09.232210294Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:116:in `block in run_callbacks'
2024-07-13T03:12:09.232213961Z from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:25:in `block in instrument'
2024-07-13T03:12:09.232217297Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications.rb:206:in `block in instrument'
2024-07-13T03:12:09.232220864Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
2024-07-13T03:12:09.232224421Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications.rb:206:in `instrument'
2024-07-13T03:12:09.232227927Z from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:35:in `instrument'
2024-07-13T03:12:09.232232405Z from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:9:in `block (2 levels) in <module:Instrumentation>'
2024-07-13T03:12:09.232264515Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
2024-07-13T03:12:09.232268864Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
2024-07-13T03:12:09.232273312Z from /usr/local/bundle/gems/activejob-traceable-0.4.2/lib/activejob/traceable/logging_patch.rb:17:in `block in tag_logger'
2024-07-13T03:12:09.232277009Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:99:in `block in tagged'
2024-07-13T03:12:09.232280496Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:37:in `tagged'
2024-07-13T03:12:09.232308759Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:99:in `tagged'
2024-07-13T03:12:09.232313377Z from /usr/local/bundle/gems/activejob-traceable-0.4.2/lib/activejob/traceable/logging_patch.rb:17:in `tag_logger'
2024-07-13T03:12:09.232317344Z from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/logging.rb:14:in `block (2 levels) in <module:Logging>'
2024-07-13T03:12:09.232338114Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
2024-07-13T03:12:09.232348653Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
2024-07-13T03:12:09.232393908Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:138:in `run_callbacks'
2024-07-13T03:12:09.232403907Z from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/enqueuing.rb:63:in `enqueue'
2024-07-13T03:12:09.232407523Z from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/enqueuing.rb:30:in `perform_later'
2024-07-13T03:12:09.232411721Z from /app/clock.rb:22:in `block in <module:Clockwork>'
2024-07-13T03:12:09.232436327Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:61:in `execute'
2024-07-13T03:12:09.232460493Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:48:in `run'
2024-07-13T03:12:09.232497852Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:136:in `block in tick'
2024-07-13T03:12:09.232504134Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `each'
2024-07-13T03:12:09.232507631Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `tick'
2024-07-13T03:12:09.232511538Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:123:in `block (2 levels) in run_tick_loop'
2024-07-13T03:12:09.232534751Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `synchronize'
2024-07-13T03:12:09.232569587Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `block in run_tick_loop'
2024-07-13T03:12:09.232574767Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:435:in `block (2 levels) in thread_block_with_current_transaction'
2024-07-13T03:12:09.232598881Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
2024-07-13T03:12:09.232621293Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:434:in `block in thread_block_with_current_transaction'
2024-07-13T03:12:09.232627715Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:140:in `block in fill_buffer': Waited 0.1 seconds (RedisClient::ReadTimeoutError)
2024-07-13T03:12:09.232631533Z from <internal:kernel>:187:in `loop'
2024-07-13T03:12:09.232635500Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:122:in `fill_buffer'
2024-07-13T03:12:09.232660216Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:114:in `ensure_remaining'
2024-07-13T03:12:09.232682678Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:85:in `getbyte'
2024-07-13T03:12:09.232720409Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/resp3.rb:113:in `parse'
2024-07-13T03:12:09.232754102Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/resp3.rb:50:in `load'
2024-07-13T03:12:09.232815587Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection.rb:94:in `read'
2024-07-13T03:12:09.232882703Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/connection_mixin.rb:31:in `call'
2024-07-13T03:12:09.232943798Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:239:in `block (2 levels) in call_v'
2024-07-13T03:12:09.232992048Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/middlewares.rb:16:in `call'
2024-07-13T03:12:09.233035799Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:238:in `block in call_v'
2024-07-13T03:12:09.233079341Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:643:in `ensure_connected'
2024-07-13T03:12:09.233120158Z from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:237:in `call_v'
2024-07-13T03:12:09.233161525Z from /usr/local/bundle/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
2024-07-13T03:12:09.233202752Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:12:in `block in call_v'
2024-07-13T03:12:09.233244220Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:19:in `block in call_with_tracing'
2024-07-13T03:12:09.233284986Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:59:in `block in with_tracing'
2024-07-13T03:12:09.233324891Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
2024-07-13T03:12:09.233365536Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:59:in `with_tracing'
2024-07-13T03:12:09.233390855Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:19:in `call_with_tracing'
2024-07-13T03:12:09.233446809Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:12:in `call_v'
2024-07-13T03:12:09.233487956Z from /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
2024-07-13T03:12:09.233542889Z from /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
2024-07-13T03:12:09.233600577Z from /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
2024-07-13T03:12:09.233647365Z from /usr/local/bundle/gems/redis-5.0.6/lib/redis/commands/scripting.rb:110:in `_eval'
2024-07-13T03:12:09.233674515Z from /usr/local/bundle/gems/redis-5.0.6/lib/redis/commands/scripting.rb:97:in `evalsha'
2024-07-13T03:12:09.233731071Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:171:in `block (2 levels) in lock'
2024-07-13T03:12:09.233771637Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:152:in `with'
2024-07-13T03:12:09.233812134Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:171:in `block in lock'
2024-07-13T03:12:09.233852428Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:208:in `recover_from_script_flush'
2024-07-13T03:12:09.233892163Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:170:in `lock'
2024-07-13T03:12:09.233931808Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `block (2 levels) in lock_instances'
2024-07-13T03:12:09.233956373Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `select'
2024-07-13T03:12:09.234014953Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `block in lock_instances'
2024-07-13T03:12:09.234055739Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:311:in `timed'
2024-07-13T03:12:09.234096165Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:257:in `lock_instances'
2024-07-13T03:12:09.234138164Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:232:in `block in try_lock_instances'
2024-07-13T03:12:09.234179010Z from <internal:numeric>:237:in `times'
2024-07-13T03:12:09.234241717Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:228:in `try_lock_instances'
2024-07-13T03:12:09.234291852Z from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:69:in `lock'
2024-07-13T03:12:09.234336304Z from /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/strategies/base.rb:24:in `lock'
2024-07-13T03:12:09.234378263Z from /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/strategies/base.rb:59:in `before_enqueue'
2024-07-13T03:12:09.234419280Z from /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/active_job_patch.rb:50:in `block (2 levels) in <module:ActiveJobPatch>'
2024-07-13T03:12:09.234459956Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:467:in `instance_exec'
2024-07-13T03:12:09.234499390Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:467:in `block in make_lambda'
2024-07-13T03:12:09.234539835Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:199:in `block (2 levels) in halting'
2024-07-13T03:12:09.234580482Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:687:in `block (2 levels) in default_terminator'
2024-07-13T03:12:09.234620116Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:686:in `catch'
2024-07-13T03:12:09.234658888Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:686:in `block in default_terminator'
2024-07-13T03:12:09.234698402Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:200:in `block in halting'
2024-07-13T03:12:09.234738137Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `block in invoke_before'
2024-07-13T03:12:09.234777751Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `each'
2024-07-13T03:12:09.234830190Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `invoke_before'
2024-07-13T03:12:09.234872608Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:116:in `block in run_callbacks'
2024-07-13T03:12:09.234912383Z from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:25:in `block in instrument'
2024-07-13T03:12:09.234953830Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications.rb:206:in `block in instrument'
2024-07-13T03:12:09.234994086Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
2024-07-13T03:12:09.235043809Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications.rb:206:in `instrument'
2024-07-13T03:12:09.235086239Z from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:35:in `instrument'
2024-07-13T03:12:09.235129650Z from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:9:in `block (2 levels) in <module:Instrumentation>'
2024-07-13T03:12:09.235170626Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
2024-07-13T03:12:09.235210852Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
2024-07-13T03:12:09.235268951Z from /usr/local/bundle/gems/activejob-traceable-0.4.2/lib/activejob/traceable/logging_patch.rb:17:in `block in tag_logger'
2024-07-13T03:12:09.235299478Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:99:in `block in tagged'
2024-07-13T03:12:09.235361023Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:37:in `tagged'
2024-07-13T03:12:09.235405837Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:99:in `tagged'
2024-07-13T03:12:09.235461101Z from /usr/local/bundle/gems/activejob-traceable-0.4.2/lib/activejob/traceable/logging_patch.rb:17:in `tag_logger'
2024-07-13T03:12:09.235503680Z from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/logging.rb:14:in `block (2 levels) in <module:Logging>'
2024-07-13T03:12:09.235529719Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
2024-07-13T03:12:09.235550358Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
2024-07-13T03:12:09.235570195Z from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:138:in `run_callbacks'
2024-07-13T03:12:09.235589130Z from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/enqueuing.rb:63:in `enqueue'
2024-07-13T03:12:09.235608587Z from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/enqueuing.rb:30:in `perform_later'
2024-07-13T03:12:09.235650204Z from /app/clock.rb:22:in `block in <module:Clockwork>'
2024-07-13T03:12:09.235709686Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:61:in `execute'
2024-07-13T03:12:09.235735484Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:48:in `run'
2024-07-13T03:12:09.235791269Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:136:in `block in tick'
2024-07-13T03:12:09.235831033Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `each'
2024-07-13T03:12:09.235871098Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `tick'
2024-07-13T03:12:09.235911334Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:123:in `block (2 levels) in run_tick_loop'
2024-07-13T03:12:09.235951168Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `synchronize'
2024-07-13T03:12:09.236004968Z from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `block in run_tick_loop'
2024-07-13T03:12:09.236048069Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:435:in `block (2 levels) in thread_block_with_current_transaction'
2024-07-13T03:12:09.236088465Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
2024-07-13T03:12:09.236128750Z from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:434:in `block in thread_block_with_current_transaction'
thanks for it @grthr I'm having a look