Explore how we can add flexibility to the storage location of project files.

[tompollard]

There is an LCP project where we would like to make data available only on BigQuery (not on PhysioNet servers or cloud buckets).

Modifying PhysioNet so that users have more flexibility in where files are hosted could be helpful more generally, e.g.

• data is too large to host on our local servers.
• specific requirements around where data can be physically hosted (e.g. only on New Zealand territory).
• benefits in terms of controlling and auditing access.

[briangow]

@tompollard , @bemoody , we should revive the discussion around this. We have identified a number of projects where we would like to have the project tables on BigQuery, but don't see the need to have the project files in a GCP bucket.

Since we have agreed that the same permission group should be used for buckets and BigQuery, we should use the same code for generating the access group. Currently to create the access_group you need to create or reference a bucket. Developing independent code for creating the access_group could be beneficial even if we continue to add files to the BigQuery dataset manually. We would probably also want to standardize how the BigQuery dataset gets named.

A separate discussion addresses the larger picture around integrating the platform with BigQuery, including the sending of files to BigQuery in an automated fashion: https://github.com/MIT-LCP/physionet-build/issues/2058 .

Also, this discussion should be considered, with respect to refactoring this code: https://github.com/MIT-LCP/physionet-build/issues/1927.

[tompollard]

(It's not at all clear from my initial post, but this issue also related to the requirement for local storage. I wanted us to be able to host data on the cloud, without a public local copy).