ipfs
filestore: update and revalidate all abspaths on re-add
This isn't super critical, as I can work around it, but I believe this would be a huge performance/usability improvement. I also hope this is the right repository for this feature.
(I believe) Filestore works like this:
- You add file, it adds hashes normally, but also stores abspath (the full filepath)
- You remove file from disk, still exists until GCd and dereferenced from MFS/pins
- You add file again from new path, if old file wasn't GCd, old abspath is still stored
- Everything breaks (if you try to access the CID, pin it, add it to MFS, pin a directory with the CID, etc, you'll get errors sometimes)
The solution I propose is to simply store the abspath for the new file as well, and when an error arises on one filepath, try the next one. If we store a status, could select the last successful abspath automatically.
Currently dealing with large directories of files, containing subdirectories and files that routinely get copied/renamed/deleted, I run into this issue all the time, where the CID I have published to IPNS fails to load certain files because their original file is gone/renamed, but the reference still exists in filestore (so the blocks need to be removed manually).
FWIW, I'm still on go-ipfs v0.7.0 for now, but filestore didn't get any changes in v0.8.0 (AFAIK).
If you want to experience this issue first-hand. Download a file to a directory. While it's downloading, add that file, and also try to files/cp it to MFS after. Then after the download finishes, try to do the same thing, but with the properly named file that's finished downloading. What happens for me usually anyways, is at least with small files, the hash ends up being the same, and the new file has an abspath pointing to something like Unfinished1043.crdownload, which doesn't exist. So when the file is opened on a browser, it loads until it times out usually.
Possible duplicate of https://github.com/ipfs/go-ipfs/issues/7465. It's not the exact same idea, but seems to be solving the same problem. I'll leave this open for now unless someone wants to close it, otherwise I'll get around to closing this later when there seems to be a clear path at least in avoiding this issue.