Geth Network Halted Due to Teku Keystore Lock Error

[gunanshuCN]

Hi, I’ve been running a Geth network successfully for 7 days, but yesterday it suddenly stopped working. On checking the logs, I found the following error in the consensus layer container, which seems to have caused the network to halt:

2024-11-05 09:46:34.274 FATAL - Failed to load keystore, error Keystore file /validator-keys/teku-keys/0x96947de9e6068c22a7716656a2755a9551b0b66c2d1a741bf84a088fe1e840e992dc39861bf8ba3e8d5b6d21e8f57e64.json.lock already in use.

Here's the YAML configuration used for the network:

  - el_type: geth
    el_image: "ethereum/client-go:v1.14.8"
    cl_type: teku
    cl_image: "consensys/teku:24.8"    
    count: 2
network_params:
  network_id: "686868"
  genesis_gaslimit: 100000000      
port_publisher:
  el:
    enabled: true
    public_port_start: 32000
  cl:
    enabled: true
    public_port_start: 33000
  vc:
    enabled: true
    public_port_start: 34000
  additional_services:
    enabled: true
    public_port_start: 35000

[parithosh]

Kurtosis is optimized for local networks that aren't long running, think CI tests or integration testing. So its very much possible that there are some issues when used otherwise. However, the kurtosis service merely uses docker volumes or kubernetes PVCs, so I don't immediately see why this issue could get triggered. I'm not sure if i have any useful fix for you!