[Feature Enhancement] Allow consumption roles to be used in shares with data.all not creating / managing policies for that role

[TejasRGitHub]

Is your idea related to a problem? Please describe. Consumption roles when registered, data.all creates a managed policy for the consumption role and if the role is data.all managed then data.all attaches the policy to the consumption role.

We have few roles which have super user permissions or are managed by the users themselves. They do not want data.all to manage the IAM policies for that role.

Describe the solution you'd like While importing a consumption role, add another check "Allow without policies" ( or some other name ) with which user can onboard a consumption role and use it as a part of share.

When the share is approved, the bucket, KMS , LF , etc will be modified but the consumption role won't be attached any policy as well as a policy won't be created. This policy creation and attachment will be done by the requestor

P.S. Don't attach files. Please, prefer add code snippets directly in the message body.

[TejasRGitHub]

Option 1

When a consumption role is data.all managed, data.all creates and attaches the IAM policy for all shares on that consumption role. On the other hand, when it is customer managed then data.all still creates the IAM policy but doesn't attach it.

Here the understanding of customer managed can be changed to reflect that for a share , everything will be handled by the customer when doing any share on the consumption role ( this includes creating and also attaching the IAM policy )

Thus, with this option, the existing database columns can be used and slight modifications have to be made ( in the share processors ) to reflect this understanding. With this, whenever a customer onboards a consumption role and selects non-data.all management then share processors will completely skip any methods ( grant_target_role_access_policy , grant_s3_iam_access , etc ) related to modifying / creating this IAM policy. Also, now when a share verifier runs, ir won't indicate that the share is unhealthy if the IAM policy is not attached as it is the responsibility of share owner to attach / make changes to their consumption roles.

Effort -

If we change the way non-data.all managed roles are treated, only share processors will have to be updated to reflect this new understanding

Cons with this approach -

With this change, existing users who have been using data.all ( version < 2.7 ) will have the previous understanding but now they have to be informed ( either via release notes, additional documentation or some custom onscreen prompt ) about this change and have to be instructed ( or atleast give a sample example ) on how they can modify their consumption role to give access to the bucket / folder in their share..

Option 2

Introduce another option in the consumption role modal , to enable user to have a non-data.all managed role with Enforce data.all policy attachment which if unchecked, data.all will completely ignore creating and even checking if the policy is attached.

In summary

Consumption Role Management ( Data.all managed )	Enforcement	Summary
Yes	N/A	Standard default case when IAM policy creation and attachment is handled by data.all
No	Yes	Here data.all will create IAM policy and make the changes for the share but won't attach it. In the share verifier health checks the share will be shown as unhealthy is the policy is not attached
No	No	Here data.all ~~will not even create IAM policy and~~ won't even enforce having IAM policy on the role. In the share verifier, healthy check won't show a share to be unheaalthy even if the policy is not attached

Effort

Database migration

1. Migration script to add another column in the consumption role table

Frontend Changes -

1. Additional toggle-able button to indicate if the user wants enforcement when indicating the role to be non-data.all managed.
2. Column in the consumption role table view to show this option for each consumption role
3. Associated graphql call changes

Backend changes

1. Changes to handle additional input of enforcement in graphql calls
2. Changes in the share processor logic based on the enforcement

Cons for this approach -

Requires more development work than option 1. Also clear understanding has to be given on the modal ( for adding consumption role ) to the user.

Option 2.1

Instead of adding another field, the data.all management can be split into 3 different options

1. data.all fully managed - Standard default case when IAM policy creation and attachment is handled by data.all
2. data.all partially managed - Here data.all will create IAM policy and make the changes for the share but won't attach it. In the share verifier health checks the share will be shown as unhealthy is the policy is not attached
3. externally managed - Here data.all won't enforce having IAM policy on the role. In the share verifier, healthy check won't show a share to be unheaalthy even if the policy is not attached.

Option 3

Instead of introducing another field in the database and creating logic around it, instead the consumption role owners can tag their IAM roles with a tag to indicate if data.all should consider attaching IAM policy to this role. Also this same can be used by share verifier to check if the IAM policy attachment check needs to be done.

For e.g. if a consumption role is tagged as "non-data.all.managed" or "externally.managed" then the share verifier while verifying the share will know that the IAM policy attachment verifications can be skipped. Similarly other tags can be created

The problem that might happen with this type of solution would be that it is highly likely that the consumption role's tag can be wiped off by either someone, by some IaC changes , etc. This will cause problems in which the user will again get share unhealthy messages even when the user didn't want those checks to happen.

[TejasRGitHub]

Implemented Solution:

Option 2.1

Consumption role management will be with the following options

1. data.all fully managed - Standard default case when IAM policy creation and attachment is handled by data.all
2. data.all partially managed - Here data.all will create IAM policy and make the changes for the share but won't attach it. In the share verifier health checks, the share will be shown as unhealthy if the policy is not attached
3. externally managed - Here data.all won't enforce having IAM policy attached on the role. In the share verifier, health check won't show a share to be unhealthy even if the policy is not attached.

Changes in the code ( Tentative )

Frontend

1. Update the UI to have dropdown with the above options. All the options will have a tool tip to explain each option in more detail.
2. data.all managed column on the consumptoin role table will contain info about this option

Backend

1. Update to the gql endpoints to accept these three option ( which will be represented as enums )
2. Share processing logic
   • While processing the share, based on the data.all management value, the policy will be attached or not attached
3. Share Verifier logic
   • Similar to the share processing, data.all management value will determine if the IAM role's policy attachment and presence has to be checked.

Migration of existing shares

An alembic script will migrate existing consumption roles to the new convention.

[TejasRGitHub]

Add Consumption role mock-up ( All the options have tool tips to explain about the policy management )

Environment Consumption Role Grid view

Share View

Tool Tip for explaining principal IAM policy management

[fourtyplustwo]

+1