cloud-platform
cloud-platform copied to clipboard
Published
20 hours ago •
ministryofjustice
ministryofjustice
Chase up teams with erroring jobs
Background
We seem to have a group of usual suspects when it comes to cronjob pods in error state.
Reach out to the owners of the common namespaces/pods and see whether they have an understanding of what is happening/whether they have a fix in place.
We should do this as a prerequisite for looking at the more cleanup solution in this ticket: https://github.com/ministryofjustice/cloud-platform/issues/5672
Example of pods:
❯ kubectl get pods -A --field-selector status.phase=Failed
NAMESPACE NAME READY STATUS RESTARTS AGE
claim-criminal-injuries-compensation-dev claim-criminal-injuries-dev-cleardown-28645988-62hsz 0/1 Error 0 10h
claim-criminal-injuries-compensation-dev claim-criminal-injuries-dev-push-to-gateway-28645988-hmq2p 0/1 Error 0 10h
claim-criminal-injuries-compensation-dev claim-criminal-injuries-dev-resubmit-failed-28645988-c5k26 0/1 Error 0 10h
claim-criminal-injuries-compensation-prod claim-criminal-injuries-prod-cleardown-28645988-qsnpp 0/1 Error 0 10h
claim-criminal-injuries-compensation-prod claim-criminal-injuries-prod-push-to-gateway-28645988-547jc 0/1 Error 0 10h
claim-criminal-injuries-compensation-prod claim-criminal-injuries-prod-resubmit-failed-28645988-566d6 0/1 Error 0 10h
court-probation-preprod analytics-data-extractor-28645980-bbrzt 0/1 Error 0 10h
court-probation-preprod analytics-data-extractor-28645980-dqhzc 0/1 Error 0 10h
court-probation-preprod analytics-data-extractor-28645980-jq4mn 0/1 Error 0 10h
court-probation-preprod analytics-data-extractor-28645980-kb598 0/1 Error 0 11h
court-probation-preprod analytics-data-extractor-28645980-lkx49 0/1 Error 0 10h
court-probation-preprod analytics-data-extractor-28645980-nn4d9 0/1 Error 0 11h
court-probation-preprod analytics-data-extractor-28645980-thg2j 0/1 Error 0 10h
hmpps-audit-dev queue-housekeeping-cronjob-28646250-6prpf 0/1 Error 0 6h27m
hmpps-audit-dev queue-housekeeping-cronjob-28646250-jnqpp 0/1 Error 0 6h22m
hmpps-audit-dev queue-housekeeping-cronjob-28646250-l7fpp 0/1 Error 0 6h27m
hmpps-audit-dev queue-housekeeping-cronjob-28646250-n4dsh 0/1 Error 0 6h26m
hmpps-audit-dev queue-housekeeping-cronjob-28646250-nct57 0/1 Error 0 6h16m
hmpps-audit-dev queue-housekeeping-cronjob-28646250-svpvp 0/1 Error 0 6h24m
hmpps-audit-dev queue-housekeeping-cronjob-28646250-x7svz 0/1 Error 0 6h27m
hmpps-audit-dev queue-housekeeping-cronjob-28646270-b4qxl 0/1 Error 0 6h13m
hmpps-audit-dev queue-housekeeping-cronjob-28646270-bdsd9 0/1 Error 0 6h15m
hmpps-audit-dev queue-housekeeping-cronjob-28646270-bthz6 0/1 Error 0 6h16m
hmpps-audit-dev queue-housekeeping-cronjob-28646270-h8ptt 0/1 Error 0 6h16m
hmpps-audit-dev queue-housekeeping-cronjob-28646270-m5wrd 0/1 Error 0 6h15m
hmpps-audit-dev queue-housekeeping-cronjob-28646270-v8tq8 0/1 Error 0 6h10m
hmpps-audit-dev queue-housekeeping-cronjob-28646270-xm8r5 0/1 Error 0 6h5m
hmpps-audit-dev queue-housekeeping-cronjob-28646280-52dl6 0/1 Error 0 6h4m
hmpps-audit-dev queue-housekeeping-cronjob-28646280-7wszh 0/1 Error 0 5h59m
hmpps-audit-dev queue-housekeeping-cronjob-28646280-k4h86 0/1 Error 0 6h5m
hmpps-audit-dev queue-housekeeping-cronjob-28646280-k7fs7 0/1 Error 0 5h54m
hmpps-audit-dev queue-housekeeping-cronjob-28646280-llk9x 0/1 Error 0 6h2m
hmpps-audit-dev queue-housekeeping-cronjob-28646280-txg7h 0/1 Error 0 6h5m
hmpps-audit-dev queue-housekeeping-cronjob-28646280-xmpbt 0/1 Error 0 6h3m
hmpps-audit-dev queue-housekeeping-cronjob-28646290-4n57p 0/1 Error 0 5h53m
hmpps-audit-dev queue-housekeeping-cronjob-28646290-8hr4t 0/1 Error 0 5h43m
hmpps-audit-dev queue-housekeeping-cronjob-28646290-8jmxh 0/1 Error 0 5h53m
hmpps-audit-dev queue-housekeeping-cronjob-28646290-9r2l6 0/1 Error 0 5h52m
hmpps-audit-dev queue-housekeeping-cronjob-28646290-b4ptn 0/1 Error 0 5h54m
hmpps-audit-dev queue-housekeeping-cronjob-28646290-vvt4z 0/1 Error 0 5h51m
hmpps-audit-dev queue-housekeeping-cronjob-28646290-xfmnf 0/1 Error 0 5h48m
hmpps-audit-dev queue-housekeeping-cronjob-28646300-28qkx 0/1 Error 0 5h42m
hmpps-audit-dev queue-housekeeping-cronjob-28646300-2f6pn 0/1 Error 0 5h42m
hmpps-audit-dev queue-housekeeping-cronjob-28646300-b4wmm 0/1 Error 0 5h37m
hmpps-audit-dev queue-housekeeping-cronjob-28646300-dl94w 0/1 Error 0 5h39m
hmpps-audit-dev queue-housekeeping-cronjob-28646300-t5s29 0/1 Error 0 5h31m
hmpps-audit-dev queue-housekeeping-cronjob-28646300-v97nq 0/1 Error 0 5h41m
hmpps-audit-dev queue-housekeeping-cronjob-28646300-xtmbz 0/1 Error 0 5h42m
hmpps-education-and-work-plan-dev export-database-to-analytical-platform-cronjob-28645980-2p8gz 0/1 Error 0 11h
hmpps-education-and-work-plan-dev export-database-to-analytical-platform-cronjob-28645980-bwvvz 0/1 Error 0 11h
hmpps-education-and-work-plan-dev export-database-to-analytical-platform-cronjob-28645980-h26ps 0/1 Error 0 11h
hmpps-education-and-work-plan-dev export-database-to-analytical-platform-cronjob-28645980-kx6r8 0/1 Error 0 11h
hmpps-education-and-work-plan-preprod export-database-to-analytical-platform-cronjob-28645980-4j5nf 0/1 Error 0 11h
hmpps-education-and-work-plan-preprod export-database-to-analytical-platform-cronjob-28645980-6w94c 0/1 Error 0 11h
hmpps-education-and-work-plan-preprod export-database-to-analytical-platform-cronjob-28645980-jdpnl 0/1 Error 0 11h
hmpps-education-and-work-plan-preprod export-database-to-analytical-platform-cronjob-28645980-kf7q8 0/1 Error 0 11h
hmpps-launchpad-dev purge-sso-requests-cronjob-28646455-4fw26 0/1 Error 0 3h11m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646455-5jz65 0/1 Error 0 3h11m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646455-6nnb9 0/1 Error 0 3h9m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646455-9mntx 0/1 Error 0 3h12m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646455-lnzqv 0/1 Error 0 3h6m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646455-xnzvs 0/1 Error 0 3h10m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646455-zmmbj 0/1 Error 0 3h
hmpps-launchpad-dev purge-sso-requests-cronjob-28646465-7wvdt 0/1 Error 0 3h
hmpps-launchpad-dev purge-sso-requests-cronjob-28646465-8kgzp 0/1 Error 0 3h
hmpps-launchpad-dev purge-sso-requests-cronjob-28646465-hfrn9 0/1 Error 0 179m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646465-hkh56 0/1 Error 0 169m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646465-kbgqc 0/1 Error 0 175m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646465-pm5xv 0/1 Error 0 3h
hmpps-launchpad-dev purge-sso-requests-cronjob-28646465-sz67c 0/1 Error 0 177m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646475-25mfk 0/1 Error 0 168m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646475-4vcsb 0/1 Error 0 163m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646475-fw4dx 0/1 Error 0 166m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646475-hlh5x 0/1 Error 0 158m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646475-jw7g6 0/1 Error 0 169m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646475-p5wcr 0/1 Error 0 169m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646475-rhwvj 0/1 Error 0 168m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646495-5g9cz 0/1 Error 0 151m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646495-d6mgh 0/1 Error 0 152m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646495-fw7t7 0/1 Error 0 151m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646495-hglck 0/1 Error 0 149m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646495-jzcls 0/1 Error 0 140m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646495-tn789 0/1 Error 0 146m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646495-x2sxr 0/1 Error 0 150m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646505-9lkxx 0/1 Error 0 129m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646505-f92kq 0/1 Error 0 137m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646505-jnh2g 0/1 Error 0 140m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646505-kgglp 0/1 Error 0 140m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646505-qthjh 0/1 Error 0 139m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646505-sz47r 0/1 Error 0 140m
hmpps-launchpad-dev purge-sso-requests-cronjob-28646505-zdbxr 0/1 Error 0 135m
hmpps-workload-preprod queue-housekeeping-cronjob-28646550-659dp 0/1 Error 0 73m
hmpps-workload-preprod queue-housekeeping-cronjob-28646550-7vjfd 0/1 Error 0 79m
hmpps-workload-preprod queue-housekeeping-cronjob-28646550-9nr98 0/1 Error 0 82m
hmpps-workload-preprod queue-housekeeping-cronjob-28646550-cg5kf 0/1 Error 0 88m
hmpps-workload-preprod queue-housekeeping-cronjob-28646550-dx4s7 0/1 Error 0 84m
hmpps-workload-preprod queue-housekeeping-cronjob-28646550-kdw8r 0/1 Error 0 87m
hmpps-workload-preprod queue-housekeeping-cronjob-28646550-mwktq 0/1 Error 0 86m
hmpps-workload-preprod queue-housekeeping-cronjob-28646565-jbssr 0/1 Error 0 71m
hmpps-workload-preprod queue-housekeeping-cronjob-28646565-js4rh 0/1 Error 0 72m
hmpps-workload-preprod queue-housekeeping-cronjob-28646565-k6m76 0/1 Error 0 67m
hmpps-workload-preprod queue-housekeeping-cronjob-28646565-kzsv4 0/1 Error 0 70m
hmpps-workload-preprod queue-housekeeping-cronjob-28646565-ltvqx 0/1 Error 0 69m
hmpps-workload-preprod queue-housekeeping-cronjob-28646565-tnzt2 0/1 Error 0 57m
hmpps-workload-preprod queue-housekeeping-cronjob-28646565-zwf5d 0/1 Error 0 63m
hmpps-workload-preprod queue-housekeeping-cronjob-28646595-cgllw 0/1 Error 0 51m
hmpps-workload-preprod queue-housekeeping-cronjob-28646595-gwv9z 0/1 Error 0 37m
hmpps-workload-preprod queue-housekeeping-cronjob-28646595-gzgbd 0/1 Error 0 50m
hmpps-workload-preprod queue-housekeeping-cronjob-28646595-hg67v 0/1 Error 0 46m
hmpps-workload-preprod queue-housekeeping-cronjob-28646595-hkcmm 0/1 Error 0 43m
hmpps-workload-preprod queue-housekeeping-cronjob-28646595-kfzdl 0/1 Error 0 51m
hmpps-workload-preprod queue-housekeeping-cronjob-28646595-kj8hs 0/1 Error 0 48m
hmpps-workload-preprod queue-housekeeping-cronjob-28646610-28zv2 0/1 Error 0 36m
hmpps-workload-preprod queue-housekeeping-cronjob-28646610-747bb 0/1 Error 0 33m
hmpps-workload-preprod queue-housekeeping-cronjob-28646610-75gm9 0/1 Error 0 31m
hmpps-workload-preprod queue-housekeeping-cronjob-28646610-ckd6b 0/1 Error 0 34m
hmpps-workload-preprod queue-housekeeping-cronjob-28646610-dq54j 0/1 Error 0 27m
hmpps-workload-preprod queue-housekeeping-cronjob-28646610-drk4c 0/1 Error 0 35m
hmpps-workload-preprod queue-housekeeping-cronjob-28646610-qg7jx 0/1 Error 0 21m
hmpps-workload-preprod queue-housekeeping-cronjob-28646625-2jtxn 0/1 Error 0 20m
hmpps-workload-preprod queue-housekeeping-cronjob-28646625-ddg7f 0/1 Error 0 6m7s
hmpps-workload-preprod queue-housekeeping-cronjob-28646625-fnlfv 0/1 Error 0 15m
hmpps-workload-preprod queue-housekeeping-cronjob-28646625-jdfnt 0/1 Error 0 12m
hmpps-workload-preprod queue-housekeeping-cronjob-28646625-jnmrv 0/1 Error 0 17m
hmpps-workload-preprod queue-housekeeping-cronjob-28646625-mpf7b 0/1 Error 0 18m
hmpps-workload-preprod queue-housekeeping-cronjob-28646625-qkjgh 0/1 Error 0 20m
hmpps-workload-preprod queue-housekeeping-cronjob-28646640-7q9n4 0/1 Error 0 2m3s
hmpps-workload-preprod queue-housekeeping-cronjob-28646640-lkzz5 0/1 Error 0 4m29s
hmpps-workload-preprod queue-housekeeping-cronjob-28646640-nvk95 0/1 Error 0 5m22s
hmpps-workload-preprod queue-housekeeping-cronjob-28646640-sdcjb 0/1 Error 0 3m26s
kuberhealthy namespace-kh-check-1718726527 0/1 Error 0 20h
laa-apply-for-legalaid-uat apply-ap-5096-sca-means-te-export-digest-28646115-54bkk 0/1 Error 0 8h
laa-apply-for-legalaid-uat apply-ap-5096-sca-means-te-export-digest-28646115-b49g8 0/1 Error 0 8h
laa-apply-for-legalaid-uat apply-main-export-digest-28646115-55897 0/1 Error 0 8h
laa-apply-for-legalaid-uat apply-main-export-digest-28646115-gs9vm 0/1 Error 0 8h
pathfinder-dev moj-data-platform-extractor-28646040-9bm9q 0/1 Error 0 10h
pathfinder-dev moj-data-platform-extractor-28646040-c7xn5 0/1 Error 0 10h
pathfinder-dev moj-data-platform-extractor-28646040-cg87r 0/1 Error 0 10h
pathfinder-dev moj-data-platform-extractor-28646040-d4ht5 0/1 Error 0 9h
pathfinder-dev moj-data-platform-extractor-28646040-fshkq 0/1 Error 0 10h
pathfinder-dev moj-data-platform-extractor-28646040-pjbdj 0/1 Error 0 10h
pathfinder-dev moj-data-platform-extractor-28646040-vns4h 0/1 Error 0 10h
pathfinder-dev push-extract-28645965-75249 0/1 Error 0 6h14m
pathfinder-dev push-extract-28645965-7cwdh 0/1 Error 0 9h
pathfinder-dev push-extract-28645965-f6vwx 0/1 Error 0 11h
pathfinder-dev push-extract-28645965-j77dn 0/1 Error 0 8h
pathfinder-dev push-extract-28645965-srg4t 0/1 Error 0 7h17m
pathfinder-dev push-extract-28645965-tgzwb 0/1 Error 0 10h
Proposed user journey
Approach
Which part of the user docs does this impact
Communicate changes
- [ ] post for #cloud-platform-update
- [ ] Weeknotes item
- [ ] Show the Thing/P&A All Hands/User CoP
- [ ] Announcements channel
Questions / Assumptions
Definition of done
- [ ] readme has been updated
- [ ] user docs have been updated
- [ ] another team member has reviewed
- [ ] smoke tests are green
- [ ] prepare demo for the team
Reference
How to write good user stories
