Files are lost for unknown reasons

[igar1991]

I upload millions of files from a few kb to a few megabytes via /bytes api. Immediately after the upload, the files are available for some time, but after a few days they cannot be retrieved.

Files are uploaded with pin enabled. Directories info

262G    ./localstore/sharky
303G    ./localstore
3.1G    ./statestore
16K     ./keys
307G    .

Even though the files become inaccessible, the size of the localstore is not reduced. Node works in light mode. Bee version: 1.7.0.

Are there any limits on the number of uploaded files through 1 node? Or is the problem something else?

[ldeffenb]

I would guess that it may be due to #3037 I have proven that locally pinned files, either via the upload or pinning APIs, can have chunks unpinned when the storage radius changes. I'm still working determining the full conditions, but I know it happens with regularity with my use case (eg totally duplicable).

I have hacked a copy of the source to increase the pin counter by 10,000 for API-directed pins and that has worked around the issue in my nodes.

[agazso]

The problem is most likely indeed the #3037.

The team is working on a fix, however this requires architectural changes to the localstore component and that means basically of a rewrite of the undelying data storage. The estimation currently is that it will be released somewhere in autumn. You can keep track of all related issues in this issue: ethersphere/bee-backlog#41

[aloknerurkar]

So with the rewrite of the localstore the pinning component has been redesigned. I will close this issue. If you find the issue persists maybe we can reopen this one or create a new one. Thanks!