gotosocial
gotosocial copied to clipboard
superseriousbusiness
[feature] Nested media storage hierarchy
Describe the bug with a clear and concise description of what the bug is.
Running a a single user instance following 304 accounts federating with 500 other instances. After four months of usage, images stopped loading in Tusky. Found this error while investigating:
Mar 13 15:51:30 gotosocial[18384]: timestamp="13/03/2023 15:51:30.102" func=transport.(*transport).do level=INFO pubKeyID=https://[user-redacted]/main-key method=GET url=https://[image-url-redacted].jpeg msg="performing request" Mar 13 15:51:30 gotosocial[18384]: timestamp="13/03/2023 15:51:30.459" func=dereferencing (*deref).populateStatusAttachments level=ERROR msg="couldn't load remote attachment https://[image-url redacted].jpeg: error writing media to storage: mkdir /vol/gotosocial/storage/01H0JNQ9XXP4WSZER54WRG7R46: too many links"
Sure enough, the storage dir has hit the upper limit for NetBSD's UFS:
Mar13:1556 gtserver:/vol/gotosocial/storage $ ls | wc -l 32766
The relevant section of config.yaml:
storage-backend: local storage-local-base-path: /vol/gotosocial/storage storage-s3-endpoint: "" storage-s3-proxy: false storage-s3-use-ssl: true storage-s3-access-key: "" storage-s3-secret-key: "" storage-s3-bucket: ""
The usual fix for this sort of thing is to increase the directory depth of the path.
What's your GoToSocial Version?
0.7.1
GoToSocial Arch
NetBSD 9.1 / amd64
What happened?
Running a a single user instance following 304 accounts federating with 500 other instances. After four months of usage, images stopped loading in Tusky. Found this problem while investigating why.
What you expected to happen?
Not hit the limit, I guess :)
The usual fix is to increase the number of intermediate directories, IE, instead of /vol/gotosocial/storage/AABBBCCCDDDD/, use /vol/gotosocial/storage/AA/BB/CCCDDD or similar.
How to reproduce it?
Run a busy enough instance on NetBSD with local storage for enough time to accumulate more than 32,000 pieces of media.
Anything else we need to know?
No response
or you could use a sensible fs with no such limits :P
but, yeah, that does seem like a sensible thing to do, and should be fairly easy to implement
or you could use a sensible fs with no such limits :P
From my 15+ years of sysadmin'ing various unix variants, I would argue 32k entries is a sensible limit, as even 10k is enough to DOS your terminal by running 'ls' ... especially over slow WAN links or when using serial consoles :)
Operating system opinions aside, I appreciate your fast response, and especially this project as a whole. Thank you for all the work.
Thanks for opening this! We'll try to get a fix + migration in soon. As a potential temporary fix, can you try running a media cleanup via the admin panel, with a max cache days value of 5 or something? That should reduce the amount of directories in your storage dir (warning: may take quite a while to run, depending on your currently configured max cache days).
A max cache days value of 7 should cover you just fine for now :). I have a very heavily federating instance, (see https://k.iim.gay) and with a max cache days of 7 I currently have 6682 entries in my local storage directory.
I've not tried running a media cleanup, but did restart my instance with the lowered max cache when I initially reported the bug. It's down to 3127 entries now, a factor of ten reduction.
Still not seeing images in Tusky, nor are there errors in the logs. In fact not seeing any HTTP traffic to request them on the nginx front end; might not be Gotosocial's problem at this point.
I've reworded this one as a feature request. I still think it's a good thing for us to do, even if it's not just for explicitly supporting UFS, but I wouldn't consider a limitation like that of a filesystem a bug in GtS. With the media cache cleanups I don't think this ends up being much of an issue in practice for anyone wanting to run GtS, even on NetBSD.
