Issue PROD-24609: Implement a secret filesystem

[Kingdutch]

Problem

The private file system is slow. This has a negative effect on the page load, especially with pages that contain many images (e.g. the stream or a dashboard page).

When loading private files Drupal is booted, the webserver, database and Redis are all busy to serve the file. This can take up to several seconds as many images form a traffic jam. If there are only 4 web workers available and 20 images are loaded you can do the math 🚦

The private file download controller invokes the file_download hook which causes the entity loads. Specifically FileAccessControllHandler::checkAccess through file_file_download but also some unoptimized Open Social hooks (e.g. social_content_export_file_download, data_policy_export_file_download which start processing before checking if it cares about the file at all; or social_automation_file_download which propagates a file download further into an event system)

Solution

The secret filesystem is a hybrid between the public and the private filesystem. If you have a valid URL then the file is publicly accessible and cacheable. However the URLs are unguessable and expire after a set amount of time. This means that only if you have access to something that can generate a URL for you do you have access to the file.

The URLs are time-bucketed which ensure that requests that occur close together get the same URL. This allows caching of files which can dramatically speed up sites (and prevents some expensive access-time checks) while still providing protection against unauthorized access.

By implementing this as a schema we can move fields over as we please and still use the private filesystem for things that may not even be shared through a link (though anyone with access to a private file can always download it to share it anyway).

Todo

• [ ] Some bikeshedding over the image path, for now this is on /system/file which isn't great (it's just /system/files - s)
• [ ] Discuss whether we re-use the private filesystem or adopt a separate folder.
• [ ] Creation of a Change Record
• [ ] Decide whether this stays in social_core or moves to a dedicated (contrib?) module
• [ ] Create update hooks to change existing configuration from private files to the secret filesystem
• [ ] Change the install configuration to use the secret filesystem by default.
• [ ] File migration.

Issue tracker

https://getopensocial.atlassian.net/browse/PROD-24609

Timestamp checking in URLs is blocked by https://www.drupal.org/project/drupal/issues/3358113

How to test

• [ ] Validate that there's sufficient PHPUnit test coverage included in the PR

Definition of done

Before merge

• [ ] Code/peer review is completed
• [ ] All commit messages are clear and clean. If applicable a rebase was performed
• [ ] All automated tests are green
• [ ] Functional/manual tests of the acceptance criteria are approved
• [ ] All acceptance criteria were met
• [ ] New features or changes to existing features are covered by tests, either unit (preferably) or behat
• [ ] Update path is tested. New hook_updates should respect update order, right naming convention and consider hook_post_update code
• [ ] Module can be safely uninstalled. Update/implement hook_uninstall and make sure that removed configuration or dependencies are removed/uninstalled
• [ ] This pull request has all required labels (team/type/priority)
• [ ] This pull request has a milestone
• [ ] This pull request has an assignee (if applicable)
• [ ] Any front end changes are tested on all major browsers
• [ ] New UI elements, or changes on UI elements are approved by the design team
• [ ] New features, or feature changes are approved by the product owner

After merge

• [ ] Code is tested on all branches that it has been cherry-picked
• [ ] Update hook number might need adjustment, make sure they have the correct order
• [ ] The Drupal.org ticket(s) are updated according to this pull request status

Release notes

Open Social has changed how it handles non-public files. Pages that contain non-public files (e.g. in images or for file downloads) will now insert those using URLs with a limited time validity. Those URLs provide protection against brute-force attacks but no longer provide access checks when requested. This allows offloading of media handling, providing a significant performance increase.

Previously these non-public files would be inserted as static long-lived URLs which required an access check for every request.

Change Record

TODO