Alden Hu

> I think we just need a proper error message instead of ICE for now And line / column of the issue? (IIUC currently we don't have them)

Was you disk ever full during the operation of the node?

The latest hotfix (aptoslabs/validator:testnet_f5d8013b0a1851da8e078394d83130d3adaf7670) fixes this

> Stamp to unblock, but consider renaming the var did it. Thanks.

Thanks @grao1991

Looks like we are bottlenecked on disk IO. Working with @geekflyer to figure out how to put in a beefy disk.

repurposing this to replay the mainnet for now.

@geekflyer @runtian-zhou how do we know these nightly runs failed? Does it post to slack?

> Are the rust changes relevant? Looking at the description I'm not seeing why we need to change the rust code? The first fixes the tool so it can replay...

```4_Run ..githubactionsreplay-verify.txt:1144:2022-10-22T05:49:56.2520579Z [main process] spawning run_1_142000000_151500000 4_Run ..githubactionsreplay-verify.txt:1145:2022-10-22T05:49:56.3000142Z [main process] spawning run_2_151500000_161000000 4_Run ..githubactionsreplay-verify.txt:1146:2022-10-22T05:49:56.3497294Z [main process] spawning run_3_161000000_170500000 4_Run ..githubactionsreplay-verify.txt:1147:2022-10-22T05:49:56.3983400Z [main process] spawning run_4_170500000_180000000 4_Run ..githubactionsreplay-verify.txt:1148:2022-10-22T05:49:56.4477032Z [main process] spawning run_5_180000000_189500000...