[Epic] Use S3 as "shared filesystem" between Cryostat, cryostat-reports, and jfr-datasource

[andrewazores]

Describe the feature

For years we have talked about and lightly explored methods we could use to share a common filesystem between these three containers, so that we would not need to use Cryostat as a gateway + proxy to open remote connections to targets and pipe JFR files across multiple network hops (target -> Cryostat -> reports or datasource). For archived recordings Cryostat opens the file from its local volume and sends it over the network to the reports generator or datasource, which is much better.

This is how it has been implemented for a long time, but there are many disadvantages:

• more points of failure for network interruptions for active recording transfers
• harder to "get it right" at each link, ie. try to optimize I/O performance by directly copying buffers, avoiding kernel/user switching, avoiding writing to disk
• different implementation of active recordings vs archived recordings is harder to maintain

Mounting a common filesystem across the containers to simply pass file path pointers around would be nice, but is technically difficult or impossible to actually achieve since all the containers need to be able to read it, Cryostat needs to be able to write it, and the reports containers are also in a different Pod.

---

Already in 3.0 I have implemented a different technique for downloading recordings: using S3 presigned URLs. For archived recordings this is obvious, because Cryostat has already opened the recording stream from a target earlier and piped that into S3 once. Now that that file is present in S3 storage we can send it to the user with a direct link to S3 so that it does not need to pass back through Cryostat. This way, we are relying on the S3 client implementation and the S3 server as much as possible, so it's much more likely that they have "got it right" for high-performance I/O than we have. (This ends up looking a lot like Cryostat 2's JWT flow recording downloads from a user's perspective)

However, downloads of Active recordings in 3.0 also follow the same implementation. When the user requests to download an active recording, rather than Cryostat opening the remote recording stream from the target and then piping that directly to the client, Cryostat instead opens the remote recording stream and uploads that to S3 as an archived recording with an expiry date and a metadata tag to indicate it isn't a user-requested archived recording. Once that upload to S3 is complete, Cryostat redirects the client to its endpoint for downloading archived recordings, and now the client follows through the exact same code path as they would for other archived recordings.

Now with that context, we can see a way to use this technique to implement the "shared filesystem" approach for cryostat-reports and jfr-datasource: instead of Cryostat sending full JFR files as POST request bodies to these services, it can instead simply send them presigned URLs from S3, the same that it would send to users for downloads. Then cryostat-reports or the datasource can download the file themselves, process it, and in the case of -reports send the response back to Cryostat. Since Cryostat 3 does not maintain its own mounted storage volume but instead relies on S3 to do this, it means Cryostat must actually re-download archived recording files from S3 and then send that file back over the network to reports or datasource again.

Not only does this lean on the S3 client and server to do more of the I/O work on Cryostat's side, it also allows the S3 server and some other HTTP implementation to do the I/O work on the reports or datasource side.

This technique does presume that the S3 implementation is able to trim the temporary/transient files generated by requests for active recordings and active reports, otherwise these will continue to accumulate over time, wasting disk space and costing the user money. Perhaps we can also implement a periodic cleanup service on Cryostat to clean these up to help ensure resources are kept trim.

Anything other information?

No response