optscale icon indicating copy to clipboard operation
optscale copied to clipboard
verified publisher icon hystax

issue in optscale pod

Open surendra-tanwar opened this issue 9 months ago • 2 comments

Image Image Image

Hello team @antoniodimariano @tm-hystax @stanfra @ek-hystax @ , All the pods have been running fine for the past 102 days, but today, a few of them suddenly went into a CrashLoopBackOff state. I'm not sure what caused this issue.

When I checked the logs using kubectl logs auth-674f46c6bb-j4gfl, I received the following error:

Error response from daemon: configured logging driver does not support reading.

I'm sharing the output of the kubectl get pods command for reference.

Please help me troubleshoot and resolve this issue.

kubectl get pods NAME READY STATUS RESTARTS AGE arcee-cfd6b495f-llpm8 0/1 CrashLoopBackOff 42 102d auth-674f46c6bb-j4gfl 0/1 Init:1/3 3 102d bi-exporter-6ffb4c485-sld5j 0/1 Init:2/4 5 102d bi-scheduler-1744642800-89q88 0/1 Completed 0 20h bi-scheduler-1744713000-bhwgl 0/1 Init:Error 0 43m bi-scheduler-1744714800-wkwv2 0/1 Init:1/2 0 13m booking-observer-scheduler-1744632780-884n6 0/1 Completed 0 23h booking-observer-scheduler-1744712820-8fstv 0/1 Init:Error 0 46m booking-observer-scheduler-1744714680-p7jwx 0/1 Init:1/2 0 15m booking-observer-worker-6f994875bf-czf9g 0/1 Init:1/3 4 102d booking-observer-worker-6f994875bf-fw7x2 0/1 Init:1/3 4 102d bulldozer-api-5c99ff64c9-r7fkf 0/1 CrashLoopBackOff 44 102d bulldozerworker-55b847f766-gjzbq 0/1 Init:1/4 3 102d bumischeduler-64b5bf75dd-lfqxx 0/1 Init:0/2 4 102d bumiworker-7bd847b6f5-t5pq4 0/1 Init:1/2 3 102d calendar-observer-scheduler-1744642800-26kpj 0/1 Completed 0 20h calendar-observer-scheduler-1744711200-lft27 0/1 Init:Error 0 73m calendar-observer-scheduler-1744714800-xq4r4 0/1 Init:0/1 0 13m calendar-observer-worker-6f86f8d756-bpnm7 0/1 Init:1/2 4 102d cleaninfluxdb-1744502400-t4dqg 0/1 Completed 0 2d11h cleanmongodb-1744642980-c88b4 0/1 Completed 0 20h cleanmongodb-1744712820-wcrph 0/1 Error 0 46m cleanmongodb-1744714620-m2fph 1/1 Running 0 15m clickhouse-0 1/1 Running 49 102d demoorgcleanup-1744416000-6jcx9 0/1 Completed 0 3d11h diproxy-f8cd6844-hlt6j 0/1 Init:1/3 3 102d diworker-6cbb68bb8b-9lz8r 0/1 Init:4/5 3 102d elk-0 1/1 Running 16 102d error-pages-66c48df767-hldr9 1/1 Running 15 102d etcd-0 0/1 ErrImageNeverPull 11 102d etcd-operator-etcd-operator-etcd-operator-5c8d485cfb-6m8jd 1/1 Running 45 102d failed-imports-dataset-generator-1744416000-b8mpz 0/1 Completed 0 3d11h gemini-scheduler-1741120800-fxknb 0/1 Completed 0 41d gemini-scheduler-1741121100-xf9zb 0/1 Init:1/3 0 41d gemini-worker-7bc654c8b9-pb95j 0/1 Init:1/3 3 102d grafana-6c64d4bb97-n55dd 2/2 Running 26 102d herald-executor-c45b89649-676pj 1/1 Running 14 102d heraldapi-959969759-l7z6f 0/1 Running 13 85d heraldengine-df7796546-2x4sx 1/1 Running 13 85d heraldengine-df7796546-lrnjd 1/1 Running 13 85d influxdb-0 1/1 Running 14 102d insider-api-5c7b9fd55f-rbst2 0/1 Init:1/3 4 102d insider-scheduler-1744416000-wp4wh 0/1 Completed 0 3d11h insider-worker-58f7585695-n4kxl 0/1 Init:1/3 3 102d jira-bus-59f9c89459-6pvjn 0/1 Init:2/4 3 102d jira-ui-687854fc9-lrfpl 0/1 Init:0/1 3 102d kataraapi-75b769c7cb-65589 0/1 Init:1/4 4 102d katarascheduler-54588d7d7d-5tss7 0/1 Init:0/1 3 102d kataraworker-57c676fff8-hklwp 0/1 Init:0/1 4 102d keeper-79f9d7f87b-jbrdr 0/1 Running 14 102d keeper-executor-55c598ffdb-4zwbn 1/1 Running 14 102d layout-cleaner-1744340400-pf88n 0/1 Completed 0 4d8h layout-cleaner-1744426800-kblnn 0/1 Error 0 3d8h live-demo-generator-scheduler-1744642800-scxjg 0/1 Completed 0 20h live-demo-generator-scheduler-1744711200-wn2zk 0/1 Error 0 73m live-demo-generator-scheduler-1744714800-znzhx 1/1 Running 0 13m live-demo-generator-worker-6748c4d7dc-ll4f5 0/1 Init:1/3 3 102d mariadb-0 1/1 Running 14 102d metroculusapi-768679694d-h9r2d 0/1 Init:1/3 4 102d metroculusscheduler-1744642800-26tht 0/1 Completed 0 20h metroculusscheduler-1744713000-m5bnj 0/1 Init:Error 0 43m metroculusscheduler-1744714800-tgccf 0/1 Init:1/3 0 13m metroculusworker-9477c8689-45qld 0/1 Init:1/5 3 102d minio-0 1/1 Running 14 102d mongo-0 1/1 Running 18 102d myadmin-6746c765f9-xpq4p 1/1 Running 14 102d ngingress-nginx-ingress-controller-d7kv7 1/1 Running 6768 104d ngingress-nginx-ingress-default-backend-7cf5b8d7f4-k56qp 1/1 Running 40 102d ngui-59c68566c-krjtx 0/1 Init:1/7 3 85d ohsu-7f5ddfbcd-ndg65 0/1 Init:0/1 3 102d organization-violations-scheduler-1744642800-zg26t 0/1 Completed 0 20h organization-violations-scheduler-1744713000-42njz 0/1 Init:Error 0 43m organization-violations-scheduler-1744714800-kbbhc 0/1 Init:1/2 0 13m organization-violations-worker-56f44c697d-mtn7p 0/1 Init:1/2 3 102d pharos-receiver-788dbc5554-d8qhl 0/1 Init:0/2 3 102d pharos-worker-5bfc77d478-hf65p 0/1 Init:0/2 3 102d power-schedule-scheduler-1744642800-htjjv 0/1 Completed 0 20h power-schedule-scheduler-1744713000-n2zch 0/1 Init:Error 0 43m power-schedule-scheduler-1744714800-8rtc6 0/1 Init:1/2 0 13m power-schedule-worker-5ff59d5f44-z44b8 0/1 Init:1/3 3 102d pre-configurator-7dxnv 0/1 Completed 0 85d rabbitmq-0 1/1 Running 14 102d redis-0 1/1 Running 14 102d report-import-scheduler-0-1744642800-wcz9t 0/1 Completed 0 20h report-import-scheduler-0-1744713000-r24vp 0/1 Init:Error 0 43m report-import-scheduler-0-1744714800-cj72q 0/1 Init:0/1 0 13m report-import-scheduler-1-1744642800-pfc24 0/1 Completed 0 20h report-import-scheduler-1-1744650000-5zhd5 0/1 PodInitializing 0 18h report-import-scheduler-24-1744416000-7tn54 0/1 Completed 0 3d11h report-import-scheduler-6-1744416000-cq5h6 0/1 Completed 0 3d11h report-import-scheduler-6-1744653600-p6v5r 0/1 Init:Error 0 17h resource-discovery-scheduler-1744642800-s9jkn 0/1 Completed 0 20h resource-discovery-scheduler-1744713000-v7rx2 0/1 Init:Error 0 43m resource-discovery-scheduler-1744714800-z56tm 0/1 Init:0/1 0 13m resource-discovery-worker-54775d56d-6qslj 0/1 Init:1/2 3 102d resource-discovery-worker-54775d56d-n4pv8 0/1 Init:1/2 3 102d resource-observer-scheduler-1744642800-jc4kv 0/1 Completed 0 20h resource-observer-scheduler-1744713000-npnbm 0/1 Init:Error 0 43m resource-observer-scheduler-1744714800-zcvh4 0/1 Init:0/1 0 13m resource-observer-worker-6f8d977485-km7kk 0/1 Init:1/2 3 102d resource-violations-scheduler-1744642800-c6l2w 0/1 Completed 0 20h resource-violations-scheduler-1744713000-64p5p 0/1 Init:Error 0 43m resource-violations-scheduler-1744714800-8kmjc 0/1 Init:0/1 0 13m resource-violations-worker-5c96f87484-tqgml 0/1 Init:1/2 3 102d restapi-68c5478d94-zxwkv 0/1 Init:1/4 4 102d risp-scheduler-1744641900-6xhzc 0/1 Completed 0 20h risp-scheduler-1744713900-4b9z7 0/1 Init:Error 0 28m risp-worker-5d6f78664c-8m9qz 0/1 Init:1/5 3 102d slacker-executor-56b5bb6b4c-7cqcx 0/1 Init:1/2 3 102d slacker-f6cbdd87f-ks7ss 0/1 Init:2/4 3 102d thanos-compactor-1744416000-bbfrf 0/1 Completed 0 3d11h thanos-query-6c8cc95cfc-7hxlg 1/1 Running 67 102d thanos-receive-0 1/1 Running 2 4h41m thanos-storegateway-0 1/1 Running 14 102d thanos-web-0 1/1 Running 14 102d trapper-scheduler-1744641900-s25wm 0/1 Completed 0 20h trapper-scheduler-1744713900-hn2rb 0/1 Init:Error 0 28m trapper-worker-5b655958cc-69srb 0/1 Init:1/5 3 102d users-dataset-generator-1744416000-2jc46 0/1 Completed 0 3d11h webhook-executor-6786d9ddc9-48bpm 0/1 Init:1/3 3 102d

surendra-tanwar avatar Apr 15 '25 11:04 surendra-tanwar

I solved this issue by upgrade optscale to latest release version.

surendra-tanwar avatar Apr 16 '25 06:04 surendra-tanwar

also please chck this ticket https://github.com/hystax/optscale/issues/537

surendra-tanwar avatar Apr 16 '25 09:04 surendra-tanwar

All issues are described in https://github.com/hystax/optscale/issues/537

stanfra avatar Apr 29 '25 08:04 stanfra