[Bug]: Node Storage pruning may not be working as expected (DEVNET,TESTNET,MAINNET)

[frankf1957]

Contact Details

No response

Description

Node Storage pruning may not be working as expected (DEVNET,TESTNET,MAINNET)

Node version

Elrond Node CLI App version v1.3.42.0-0-47b8513d6a/go1.17.6/linux-amd64/b3f0d25a83

Host machine

Hetzner - dedicated

Steps to reproduce

I received a disk space utilization alert on my DEVNET system over the weekend. Investigation showed that Epoch_nnnnn directories we not being cleaned (removed) on my DEVNET nodes. There were 98 Epochs for each DEVNET validator node. I checked my TESTNET system and noticed the same behavior.

I just looked on Mainnet this morning and see that Mainnet seems to be affected the same. (although the count of Epoch_nnn directories is not so high yet).

All nodes are validators, I have no observers.

On which network is the bug manifesting itself?

Mainnet

Relevant log output

mainnet
=======

Elrond Node CLI App version v1.3.42.0-0-47b8513d6a/go1.17.6/linux-amd64/b3f0d25a83

$ elrond-nodes/node-0$ cat config/config.toml | egrep -v '^$|#' | sed -n '/\[StoragePruning\]/,/^\[/p'
[StoragePruning]
    Enabled = true
    ValidatorCleanOldEpochsData = true
    ObserverCleanOldEpochsData = false
    AccountsTrieCleanOldEpochsData = true
    AccountsTrieSkipRemovalCustomPattern = "%50"
    NumEpochsToKeep = 4
    NumActivePersisters = 3
    FullArchiveNumActivePersisters = 10
[MiniBlocksStorage]

$ elrond-nodes/node-0$ ls -ld db/1/Epoch_*
drwx------ 3 elrond elrond 4096 Sep 18 15:18 db/1/Epoch_780
drwx------ 3 elrond elrond 4096 Sep 19 15:18 db/1/Epoch_781
drwx------ 3 elrond elrond 4096 Sep 20 15:18 db/1/Epoch_782
drwx------ 3 elrond elrond 4096 Sep 21 15:18 db/1/Epoch_783
drwx------ 3 elrond elrond 4096 Sep 22 15:18 db/1/Epoch_784
drwx------ 4 elrond elrond 4096 Sep 26 16:07 db/1/Epoch_785
drwx------ 4 elrond elrond 4096 Sep 26 16:07 db/1/Epoch_786
drwx------ 4 elrond elrond 4096 Sep 26 16:07 db/1/Epoch_787
drwx------ 4 elrond elrond 4096 Sep 26 16:07 db/1/Epoch_788
drwx------ 3 elrond elrond 4096 Sep 27 15:19 db/1/Epoch_789
drwx------ 3 elrond elrond 4096 Sep 28 15:19 db/1/Epoch_790
drwx------ 3 elrond elrond 4096 Sep 29 15:19 db/1/Epoch_791
drwx------ 3 elrond elrond 4096 Sep 30 15:19 db/1/Epoch_792
drwx------ 3 elrond elrond 4096 Oct  1 15:19 db/1/Epoch_793
drwx------ 3 elrond elrond 4096 Oct  2 15:19 db/1/Epoch_794

ls -ld db/1/Epoch_* | wc -l
15


devnet
======

Elrond Node CLI App version D1.3.43.0-0-3b2f905ab2/go1.17.6/linux-amd64/be36dd5835

elrond-nodes/node-0$ cat config/config.toml | egrep -v '^$|#' | sed -n '/\[StoragePruning\]/,/^\[/p'
[StoragePruning]
    Enabled = true
    ValidatorCleanOldEpochsData = true
    ObserverCleanOldEpochsData = false
    AccountsTrieCleanOldEpochsData = true
    AccountsTrieSkipRemovalCustomPattern = "%50"
    NumEpochsToKeep = 4
    NumActivePersisters = 3
    FullArchiveNumActivePersisters = 10
[MiniBlocksStorage]

elrond-nodes/node-0$ ls -ld db/D/Epoch_*
drwx------ 3 elrond elrond 4096 Oct  1 12:43 db/D/Epoch_2231
drwx------ 4 elrond elrond 4096 Oct  1 21:08 db/D/Epoch_2232
drwx------ 4 elrond elrond 4096 Oct  1 21:08 db/D/Epoch_2233
drwx------ 4 elrond elrond 4096 Oct  1 21:08 db/D/Epoch_2234
drwx------ 4 elrond elrond 4096 Oct  1 21:08 db/D/Epoch_2235
drwx------ 3 elrond elrond 4096 Oct  1 22:43 db/D/Epoch_2236
drwx------ 3 elrond elrond 4096 Oct  2 00:43 db/D/Epoch_2237
drwx------ 4 elrond elrond 4096 Oct  2 09:00 db/D/Epoch_2238
drwx------ 4 elrond elrond 4096 Oct  2 09:00 db/D/Epoch_2239
drwx------ 4 elrond elrond 4096 Oct  2 08:59 db/D/Epoch_2240
drwx------ 4 elrond elrond 4096 Oct  2 09:00 db/D/Epoch_2241
drwx------ 3 elrond elrond 4096 Oct  2 10:44 db/D/Epoch_2242
drwx------ 4 elrond elrond 4096 Oct  2 19:02 db/D/Epoch_2243
drwx------ 4 elrond elrond 4096 Oct  2 19:02 db/D/Epoch_2244
drwx------ 4 elrond elrond 4096 Oct  2 19:02 db/D/Epoch_2245
drwx------ 4 elrond elrond 4096 Oct  2 19:02 db/D/Epoch_2246
drwx------ 3 elrond elrond 4096 Oct  2 20:45 db/D/Epoch_2247
drwx------ 3 elrond elrond 4096 Oct  2 22:45 db/D/Epoch_2248
drwx------ 3 elrond elrond 4096 Oct  3 00:45 db/D/Epoch_2249
drwx------ 3 elrond elrond 4096 Oct  3 02:45 db/D/Epoch_2250
drwx------ 3 elrond elrond 4096 Oct  3 04:45 db/D/Epoch_2251
drwx------ 3 elrond elrond 4096 Oct  3 06:45 db/D/Epoch_2252
drwx------ 3 elrond elrond 4096 Oct  3 08:45 db/D/Epoch_2253
drwx------ 3 elrond elrond 4096 Oct  3 10:45 db/D/Epoch_2254
drwx------ 3 elrond elrond 4096 Oct  3 12:45 db/D/Epoch_2255

elrond-nodes/node-0$ ls -ld db/D/Epoch_* | wc -l
25

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

[bogdan-rosianu]

Hello! Thanks for reporting this. Can you also attach logs with log level *:DEBUG?

[bogdan-rosianu]

catched and fixed in #4549

[bogdan-rosianu]

@frankf1957 can you confirm that the issue is fixed and proceed with closing this issue?

[frankf1957]

Checked and fixed in all environments. No further action is required. Closing issue.