Replace failing disk in Kessie (reprise)
Kessie has another failing disk.
Device: /dev/sg1 [cciss_disk_02] [SCSI], SMART Failure: HARDWARE IMPENDING FAILURE GENERAL HARD DRIVE FAILURE
Device info:
[HP MB2000FAMYV HPD7], lu id: 0x5000c50034133f93, S/N: 9WM5CW4800009137GXTK, 2.00 TB
The disks are getting very old and seem to be easily prone to failure during high workloads e.g.: Disk intensive re-processing. Alternative might be to replace the machine: https://github.com/openstreetmap/operations/issues/406
I likely have a spare disk available. I also have a much newer 4TB non-HP firmware SAS disk available.
Ended up not being able to find a suitable caddie. Disk ordered and being shipped direct to data centre. Will update data centre once tracking number is known.
Disk has been posted. It should arrive tomorrow. The hosting provider has been notified and will install per their schedule.
Disk was replaced today. RAID array is rebuilding.