osb-cmdb Support sharing service instances across osb client platforms/tenants

Expected behavior

As an app developer

in order to use the same service instance (say a kafka cluster topic, or a log collection service) across apps deployment in distinct platforms (say CF1 and K8S 2)
I need to be able to share a service instance across osb clients

Observed behavior

Osb API does not support service instance sharing

Alternatives UXs

Sharing opt-in with osb-cmdb tenants

In this UX, an owner of an existing service instance grants a specific 3rd party osb client permission to access a share of its service instance. When a 3rd party service instance then asks to create the service instance with the specified instance correlation key, then a share is used instead

This is a bit similar to CF service instance sharing UX

Step 1: service owner: specifies accepted sharing requests

save the following json in authZ.jsonspecifying sharing authorization

{
  "x_osb_cmdb": {
    "sharing_to": [
      {
        "tenant": "cf1",
        "profile": "cloudfoundry",
        "org_name": "my_org",
        "space_name": "my_space",
        "org_guid": "my_org_guid",
        "space_guid": "my_space_guid",
        "instance_name": "my_instance"
      },
      {
        "tenant": "ks2",
        "profile": "kubernetes",
        "namespace": "my_namespace",
        "originating_identity_groups": "[\"system:serviceaccounts\",\"system:serviceaccounts:catalog\",\"system:authenticated\"]",
        "originating_identity_username": "system:serviceaccount:catalog:service-catalog-controller-manager",
        "service_instance_guid": "a-service-instance-guid"
      }
    ]
  }
}

cf create-service-instance mysql 10mb mydb -c authZ.json or cf update-instance mydb -c authZ.json

Details about the sharing_to structure:

each field defines a sharing constraint
all sharing constraints need to match for a lease request to be accepted
some sharing constraints are mutually exclusive: *-name and *-guid. When both are specified (e.g. space-name and space-guid) then *-name (e.g. space-name) are ignored and are considered as documentation commenting the value of *-guid constraints.
- *-name constraints may be used among trusted tenants where there is low risk that an attacker can forge a space/org of the same name to obtain a lease without being granted actual permissions. Names are easier to reason with but grant a lower security.
- *-guid constraints are used among less trusted tenants, where a service owner needs to actually control exactly the org/space of the leaser (as guid are immutable and hard to inject into osb client platforms). Guids are harder to reason on

Step 2: service leaser: instanciate service instance lease

A distinct user provisions a service instance with all of the matching criteria (tenant, org/space/instance), plus service offering type

# cf login to cf1
cf t -o my_org -s my_space
cf create-service-instance mysql 10mb mydb-share

The status message indicates this is a shared instance (aka a lease)

cf service mydb-share
Showing info of service gberche-pmysql in org diod-gcp-demo-sosh-care / space front-end as gberche...

name:             mydb-share
service:          p-mysql
tags:             
plan:             10mb
description:      MariaDB databases 10.3.22 on demand on shared cluster
documentation:    https://mariadb.com/kb/en/library/
dashboard:        https://[..]
service broker:   p-mysql

Showing status of last operation from service mydb-share...

status:    create succeeded
message:   This is a lease from instance in tenant 1 with guid=1234
started:   2020-12-17T15:53:15Z
updated:   2020-12-17T15:53:15Z

Step 2b: service owner: review list of leases

The service owner can obtain a list of leases from a shared service instance:

$ cf service-instance mydb --params
{
<initial service specific params omited >,
  "x_osb_cmdb": {
    "sharing-to":[ ...],
    "leases": [
      {
        "tenant": "cf1",
        "profile": "cloudfoundry",
        "org_name": "my_org",
        "space_name": "my_space",
        "org_guid": "my_org_guid",
        "space_guid": "my_space_guid",
        "instance_name": "my_instance",
        "instance_guid": "my_instance-guid"
      },
      {
        "tenant": "cf2",
        "profile": "cloudfoundry",
        "org_name": "my_org",
        "space_name": "my_space",
        "org_guid": "my_org_guid",
        "space_guid": "my_space_guid",
        "instance_name": "my_instance",
        "instance_guid": "my_instance-guid"
      }
    ]
  }
}

#### Step 3: service owner: fails to delete a shared service instance

The service owner can not delete a shared service instance when there are remaining leases

$ cf delete-service-instance -f mydb FAILED Broker Error: service instance can not deleted as it is still shared


[ ] TODO: consider displaying shares in the error message `Broker Error: service instance can not deleted as it is still shared with [ {  "tenant": "cf1",        "profile": "cloudfoundry",        "org_name": "my_org",        "space_name": "my_space",       "org_guid": "my_org_guid",         "space_guid": "my_space_guid",        "instance_name": "my_instance" }`

#### Step 4a: service owner: revokes service sharing permissions

The service owner revokes the service sharing permissions which prevents new leases from being created. 

Note that revoking sharing permissions does not delete leases as there is no notification mechanism available to notify platforms of deletion of a shared service instance in order to delete leases.

Save the following json in `authZ.json` specifying sharing authorization:

```json
{
  "x_osb_cmdb": {
    "sharing_to": [    ]
  }
}

cf update-instance mydb -c authZ.json

Step 4b: service leaser: delete lease

The service leaser can always delete a lease

# cf login to cf1
cf t -o my_org -s my_space
cf delete-service-instance -f mydb-lease

Step 5: service owner: unprovision share without leases

The service owner can delete its shared service instance when there are no associated leases

$ cf delete-service-instance -f mydb 
OK

Step 5a: service owner: purge leases

The service owner can purge any leases associated to a shared service instance:

$ cf update-service-instance mydb -c
{
<initial service specific params omited >,
  "x_osb_cmdb": {
    "sharing-to":[ ...],
    "leases": [ // one or all leases removed from here
    ]
  }
}

Step 5b: service owner: unprovision share and purge leases

In case the service owner lost control over the leases, he can delete a shared service instance and purging any associated leases. This creates dangling orphan leases that will need to be purged manually by service leasers

Save the following json in authZ.json specifying sharing authorization:

{
  "x_osb_cmdb": {
    "sharing_to": [    ],
    "purge_leases_on_delete": "true" 
  }
}

$ cf update-instance mydb -c authZ.json 
OK
$ cf delete-service-instance -f mydb 
OK

Explicit lease creation using sharing token exchanged out-of-bands

In this UX, an owner of an existing service instance obtains a sharing secret that it provides out-of-bands to a service leaser. The service leaser uses the sharing token to create a lease before the token expiration date.

Step 1: service owner: obtains sharing token

The service owner looks up a sharing token that it may send to another team or use himself to create a service lease

$ cf service mydb --params
{
  "x_osb_cmdb": {
    "sharing_token": 
      {
        "token": "123456...",
        "expiration-date": "Mon, 22 Mar 2021 17:42:55 +0100"
      }
     // other osb-cmdb backend service instance params omited
  },
// other brokered service instance params omited
}

$ cf service mydb --params | jq -r .x_osb_cmdb.sharing_token.token
123456...

Step 2: service owner: obtains sharing token

A distinct user provisions a service instance matching the same service offering type and sharing token

# cf login to any other platform
# cf login to cf1
cf create-service-instance mysql 10mb mydb-share -c '{"sharing_token":"123456..."}

The status message indicates this is a shared instance (aka a lease)

cf service mydb-share
Showing info of service gberche-pmysql in org diod-gcp-demo-sosh-care / space front-end as gberche...

name:             mydb-share
service:          p-mysql
tags:             
plan:             10mb
description:      MariaDB databases 10.3.22 on demand on shared cluster
documentation:    https://mariadb.com/kb/en/library/
dashboard:        https://[..]
service broker:   p-mysql

Showing status of last operation from service mydb-share...

status:    create succeeded
message:   This is a lease from instance in tenant 1 with guid=1234
started:   2020-12-17T15:53:15Z
updated:   2020-12-17T15:53:15Z

Step 2b: service owner: review list of leases

The service owner can obtain a list of leases from a shared service instance:

$ cf service-instance mydb --params
{
<initial service specific params omited >,
  "x_osb_cmdb": {
    "sharing_token":[ ...],
    "leases": [
      {
        "tenant": "cf1",
        "profile": "cloudfoundry",
        "org_name": "my_org",
        "space_name": "my_space",
        "org_guid": "my_org_guid",
        "space_guid": "my_space_guid",
        "instance_name": "my_instance",
        "instance_guid": "my_instance-guid"
      },
      {
        "tenant": "cf2",
        "profile": "cloudfoundry",
        "org_name": "my_org",
        "space_name": "my_space",
        "org_guid": "my_org_guid",
        "space_guid": "my_space_guid",
        "instance_name": "my_instance",
        "instance_guid": "my_instance-guid"
      }
    ]
  }
}

#### Step 3: service owner: fails to delete a shared service instance

The service owner can not delete a shared service instance when there are remaining leases

$ cf delete-service-instance -f mydb FAILED Broker Error: service instance can not deleted as it is still shared


[ ] TODO: consider displaying shares in the error message `Broker Error: service instance can not deleted as it is still shared with [ {  "tenant": "cf1",        "profile": "cloudfoundry",        "org_name": "my_org",        "space_name": "my_space",       "org_guid": "my_org_guid",         "space_guid": "my_space_guid",        "instance_name": "my_instance" }`


#### Step 5: service owner: unprovision share without leases

The service owner can delete its shared service instance when there are no associated leases

$ cf delete-service-instance -f mydb OK


#### Step 5a: service owner: purge leases

The service owner can purge any leases associated to a shared service instance:

$ cf update-service-instance mydb -c { , "x_osb_cmdb": { "sharing-to":[ ...], "leases": [ // one or all leases removed from here ] } } OK



#### Step 5b: service owner: unprovision share and purge leases

In case the service owner lost control over the leases, he can delete a shared service instance and purging any associated leases. This creates dangling orphan leases that will need to be purged manually by service leasers


Save the following json in `authZ.json` specifying sharing authorization:

```json
{
  "x_osb_cmdb": {
    // other params omitted here
    "purge_leases_on_delete": "true" 
  }
}

$ cf update-instance mydb -c authZ.json 
OK
$ cf delete-service-instance -f mydb 
OK

The following table summarizes key differences/implications among the alternative UXs

Actor	Use case	Sharing opt-in	Sharing token
Owner	Control where leases can be created	Explicit via lease constraints	Out-of-band exchange of sharing token with leaser(s)
Owner	Audit traces of granting leases	Platform audits of assigned lease constraints	Hard to control the dissemination of the sharing token
Leaser	Create lease	Implicit, no additional action	Explicit by specifying sharing token before expiration time

Under the hood, osb-cmdb uses:

metadata assigned to service instance to store sharing permissions
provisions shared-service instance when permissions match

Service broker involvements

catalog opt-out or opt-in: a service broker may indicate support for sharing using metadata.shareable in the catalog endpoint
service instance sharing validation: a service broker may choose to accept or reject the creation of a new lease when responding to the PATCH /v2/service_instances/:instance_id endpoint triggered by osb-cmdb (as an equivalent to cf update-service mydb -c '{"x_osb_cmdb": {"leases": [ ...]} }')
- e.g. a data-service deployed only in a eu-1a az may refuse to share a service instance with clients in azs eu-2*
brokered service binding metadata (see #104): a broker may assign distinct binding permissions based on instance sharing see enable-sharing.html#binding-permissions

the service broker may return credentials with different permissions depending on which client platform/space an app is bound from. For example, a messaging service may permit writes from the originating client platform/space and only reads from any client platforms/spaces that the service is shared into. To determine whether the space of the app is the same as the originating space of the service instance, the service broker can compare the context.space_guid and bind_resource.space_guid fields in the brokered binding request available in the metadata propagated by the osb-cmdb

Affected release

Reproduced on version x.y -->

Dec 17 '20 16:12 gberche-orange

Pending improvements/fixes to the proposed UX above:

opt-in:
- [x] clarify optional/mutually-exclusive fields in sharing-to structure, and input validation algorithm
- [ ] Prototype UX using underlying CF service instance sharing to confirm feasibility, including async and message UX
- [ ] Leverage OSB API 2.17 Service instance metadata to surface sharing status to users
- [x] Leverage OSB API 2.16 Service instance fetching and up-coming CF v8 support to surface sharing status to users
- [ ] Add support for refreshing the leases upon updates/mutations on the shared instance (service plan, service params)
  - [ ] Suggest polling/refresh of service instance in https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#fetching-a-service-instance which is currently not supported

Otherwise, Platforms SHOULD NOT attempt to call this endpoint under any circumstances.

By polling this endpoint, Platforms MAY refresh their view of a successfully provisioned Service Instance, and discover > out-of-bands changes that were applied on a Service Instance (such as parameters, dashboard url or maintenance info).

other alternatives:
- [ ] consider alternatives to opt-in
  - [x] Explicit lease creation using sharing token exchanged out-of-bands

Feb 25 '21 08:02 gberche-orange

Hi Guillaume,

After reading this, here is some feedback.

When specifying a sharing authorisation, specifying both Org name and Org GUID might cause confusion whenever they don't match. The same applies for Space name and GUID.

{
  "x-osb-cmdb": {
    "sharing-to": [
      {
        "tenant": "cf1",
        "profile": "cloudfoundry",
        "org-name": "my-org",
        "space-name": "my-space",
        "org-guid": "my-org2-guid",
        "space-guid": "my-space2-guid",
        "instance-name": "my-instance"
      },
      ...

Whenever an Org (or Space) would be renamed, then the Org names/GUIDs specified in authZ.json specs would not match anymore.

In case inconsistent Org name & GUID is supported in authZ.json, this specification would benefit from detailing the algorithm for resolving the exact target Org. Or, just choose one of the fields to rely on. Names are for humans, whereas GUIDs are for machines. Humans need readable things, and machine needs precise references.

Whenever names would be preferred, then the actual GUIDs should be resolved by the broker in some way so that the system is resistant to Orgs renames. This resolving could be made through a read-only access to the target tenant, or through a unique “meta” resolver API that would have read-only access to all tenants.

Whenever GUIDs would be preferred, then humans would need some easy way to convert them into understandable names, just for the sake of verifying that the parameters are correct (useful for debugging).

Best, Benjamin

Mar 21 '21 10:03 bgandon

thanks @bgandon for your feedback

I refined the sharing optin ux with the following details. Please let me know if this makes sense to you.

Details about the sharing_to structure:

each field defines a sharing constraint

all sharing constraints need to match for a lease request to be accepted

some sharing constraints are mutually exclusive: *-name and *-guid. When both are specified (e.g. space-name and space-guid) then *-name (e.g. space-name) are ignored and are considered as documentation commenting the value of *-guid constraints.

*-name constraints may be used among trusted tenants where there is low risk that an attacker can forge a space/org of the same name to obtain a lease without being granted actual permissions. Names are easier to reason with but grant a lower security.

*-guid constraints are used among less trusted tenants, where a service owner needs to actually control exactly the org/space of the leaser (as guid are immutable and hard to inject into osb client platforms). Guids are harder to reason on

Mar 22 '21 17:03 gberche-orange

Suggested refinement by @poblin-orange

surface a distinct service offering "service proxy" for leases with the following benefits
- avoids stale cached data by client leasers

May 11 '22 15:05 gberche-orange