humanmade
Multisite: wp-admin timeout for some network sites after WordPress 6/PHP 8 upgrade
We recently pushed a couple updates:
- WordPress 5.9.3 -> 6.1.1
- PHP 7.4 -> 8.1
After which wp-admin started timing out on 3-4 of the sites on our multisite network with around 50 sites. This happened with all other plugins disabled except for s3-uploads.
Looking through the AWS X-Ray traces, it appears to be happening when WordPress looks up the available total upload space for a given site with get_upload_space_available():
The above screenshot is from a site that isn't affected by the timeout issue, since the X-Ray traces for affected sites get stuck in a "pending" state without any segments... Those requests time out after 60s without any information in the logs. I haven't tried increasing the 60s timeout to see if it would eventually complete.
At first we thought maybe it had something to do with the number of media objects the affected sites had. Total objects for these sites range from around 1500-15000. But the trace above is from a site with 4000 objects and still works.
As a temporary workaround, disabling Site upload space in network settings seems to have resolved the issue. But I'm worried that the bug will be triggered by some other operation that uses S3_Uploads\Stream_Wrapper::dir_readdir.
Querying the S3 CloudTrail logs shows thousands of ListObjects events when the timeouts occur. Maybe an infinite loop?
For comparison, the site with ~4000 objects that doesn't time out made less than 100 ListObjects calls in all my testing.
Hmm I'm not sure what we should do here. S3 Doesn't have a good way to calculate space used with a given prefix. So I'm not sure we can support the Site upload space option, unless we do the "accounting" which seems pretty error prone. IMO we'd be better to hardcode the upload_space_check_disabled option to true so this isn't allowed.
I think this is best left in the control of those using S3 Uploads rather than within the plugin itself?
@rmccue upload_space_check_disabled isn't easily toggled by the user. AFAIK there isn't a UI for that option, and I believe on multisite it needs to be activated individually for every site (I could be wrong though).
If there was a way for s3-uploads to force it to true for all sites that would be a better solution I think.
Either way it should probably be documented as a known issue.
This can be filtered with add_filter( 'pre_site_option_upload_space_check_disabled', '__return_false' ) in your code. The question is whether we add that filter to S3 Uploads by default.
I could imagine this functionality may be in use by some users, so I'm not sure if it's a good default for us to include, hence my suggestion that it could be better for users to add this rather than us. (When I say "user", I don't necessarily mean end-users, I mean those who are installing S3 Uploads.) That said, as @joehoyle says there's fundamentally no way for us to do this in a performant way so we may be best off forcing it off.
