Mirror Node not indexing new transactions - RecordFileDownloader - None of the data files could be verified

[mmalik-al]

Node started producing following errors and at the same time none of the new transactions were indexed on the mirror node (It was checked on Token create and mint)

2025-01-17T12:44:00.889Z  WARN scheduling-4 c.h.m.i.d.r.RecordFileDownloader Failed processing signature from node 0 corresponding to 

2024-12-18T13_16_06.101406957Z.rcd_sig. Will retry another node: Running hash mismatch for file 2024-12-18T13_16_06.101406957Z.rcd.gz. Expected = a5843cc0cabf0555fde2a04da54454af5b61d41cdb949310b6e7c24b42edf6dafc94566be813f0a8c682f261c6829075, Actual = a7a5a84cf87a88b50619825c367b58a80fd352da4f6bdbe9d4a9c1178087aad467116c0848512daeccc9b75bbd2dc4f8 

2025-01-17T12:44:00.889Z ERROR scheduling-4 c.h.m.i.d.r.RecordFileDownloader None of the data files could be verified, signatures: [StreamFileSignature(filename=2024-12-18T13_16_06.101406957Z.rcd_sig, node=0, signatureType=SHA_384_WITH_RSA, status=CONSENSUS_REACHED, streamType=RECORD, version=6)] 

2025-01-17T12:44:00.891Z  WARN scheduling-4 c.h.m.i.d.r.RecordFileDownloader Failed processing signature from node 0 corresponding to 

2024-12-18T13_16_07.041323305Z.rcd_sig. Will retry another node: Running hash mismatch for file 2024-12-18T13_16_07.041323305Z.rcd.gz. Expected = a5843cc0cabf0555fde2a04da54454af5b61d41cdb949310b6e7c24b42edf6dafc94566be813f0a8c682f261c6829075, Actual = 0ad8d509ffbc7faafb103a752374904cafbe59d795fb788f943fc4b198d55ac2c9c0be39de9bd74e5d4495a930399cef 

2025-01-17T12:44:00.891Z ERROR scheduling-4 c.h.m.i.d.r.RecordFileDownloader None of the data files could be verified, signatures: [StreamFileSignature(filename=2024-12-18T13_16_07.041323305Z.rcd_sig, node=0, signatureType=SHA_384_WITH_RSA, status=CONSENSUS_REACHED, streamType=RECORD, version=6)] 

2025-01-17T12:44:00.892Z  WARN scheduling-4 c.h.m.i.d.r.RecordFileDownloader Failed processing signature from node 0 corresponding to 
2024-12-18T13_16_08.123589695Z.rcd_sig. Will retry another node: Running hash mismatch for file 2024-12-18T13_16_08.123589695Z.rcd.gz. Expected = a5843cc0cabf0555fde2a04da54454af5b61d41cdb949310b6e7c24b42edf6dafc94566be813f0a8c682f261c6829075, Actual = abc010e486d80bdcd5dd3908cb9c02e64a3d0eea75315c59005cf645e4ba47fa9750ea6143dc17b238f1783be9c0d0df 

2025-01-17T12:44:00.892Z ERROR scheduling-4 c.h.m.i.d.r.RecordFileDownloader None of the data files could be verified, signatures: [StreamFileSignature(filename=2024-12-18T13_16_08.123589695Z.rcd_sig, node=0, signatureType=SHA_384_WITH_RSA, status=CONSENSUS_REACHED, streamType=RECORD, version=6)] 

What were the steps that caused this state? (This is not 100% certain replication steps but actions that we did that lead to those errors):

1. Clone repository and run it for the first time
2. Change docker compose file to push file streams to the google Pub/Sub instead of local database (importer service)
3. Change node to the default config
4. Node stopped indexing new transactions

[madebymatty]

Also having the same issues, with the network node also sometimes going down.

Restart clears all data, same as stop