nft.storage
nft.storage copied to clipboard
Max pending/unpinned quota
We've recently had a user pin hundreds of thousands of CIDs that they do not have access to and can never retrieve.
This adds load to our cluster and dagcargo that is currently never resolved.
The idea here is to add a quota for maximum "in flight" pins. This effectively puts a per user cap on the number of pins that can be submitted that never become pinned. The user then starts to receive 4xx errors for pin requests, which should prompt them to delete bad pins. It also serves as a crude rate limiting mechanism.
Alternatively an adequete cleanup job could take care of this, but the idea here is to reduce that load by encouraging users to clean up after themselves.
Note - this should either be limited to PSA only, or at least have different PSA / non-PSA limits. The PSA ones are prone to effectively-abuse, while the upload-based ones are somewhat symmetric.
Looking into this, I think the best approach is to add a route middleware to routes POST /pins and /pins/:requestId, augmenting the authorization layer to also check the on going requests. We can have ENV variable with max amount