service-broker icon indicating copy to clipboard operation
service-broker copied to clipboard

Published 20 hours ago • verified publisher icon appscode

Support for KubeDB service provisioning via service-broker in Cloud Foundry environment

Open bikramnehra opened this issue 5 years ago • 21 comments

I am trying to provision databases using KubeDB via service-broker in SUSE Cloud Foundry environment and till now seen success in registering the broker with cf and listing sevices/plans being offered by KubeDB:

$ cf service-brokers
Getting service brokers as admin...

name                      url
appscode-service-broker   https://appscode-service-broker.appscode.svc
persi-nfs-broker          http://nfs-broker-nfsbroker.scf.svc.cluster.local:8999

$ cf marketplace
Getting services from marketplace in org system / space space as admin...
OK

service   plans               description            broker
mysql     demo-mysql, mysql   KubeDB managed MySQL
redis     demo-redis, redis   KubeDB managed Redis

However, I am facing issues while trying to provision services:

cf create-service mysql demo-mysql mysql-example-svc

This is the response code from the broker:

$ cf create-service mysql demo-mysql mysql-example-svc
Creating service instance mysql-example-svc in org system / space space as admin...
Unexpected Response
Response code: 500
CC code:       0
CC error code:
Request ID:    ec6f0b5a-7d5c-4c29-6d2f-ba3ae586333d::1de25a10-852c-442c-a3eb-ceb01387ebb6
Description:   {
  "description": "HTTPClient::KeepAliveDisconnected: ",
  "error_code": "CF-HttpRequestError",
  "code": 10001,
  "http": {
    "uri": "https://appscode-service-broker.appscode.svc/v2/service_instances/6963454c-976b-422a-ba24-4ebd25c692d8?accepts_incomplete=true",
    "method": "PUT"
  }
}

FAILED

Here's the link to stack trace from the broker pod.

By looking at the logs it seems like a null exception: Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x1ac0c80), concrete:(*runtime._type)(nil), asserted:(*runtime._type)(0x1a564a0), missingMethod:""} (interface conversion: interface {} is nil, not string)

I believe there are attributes expected by the broker which are not being passed to it in the request. If the support for such attributes is present then it needs to be surfaced. Normally, in a cf env service specific config parameters can be provided in json object:

http://cli.cloudfoundry.org/en-US/cf/create-service.html

We might also want to look at minibroker to see how CloudFoundry support was added. Specifically the part where it mentions about cf cloud controller handling request for services instead of service catalog.

bikramnehra avatar Feb 27 '19 05:02 bikramnehra

I am seeing the same issue when trying to make this request directly using curl e.g: I am able to successfully list catalog services, using the following request:

curl https://appscode-service-broker.appscode.svc/v2/catalog --resolve 'appscode-service-broker.appscode.svc:443:<IP>' --cacert cacert.crt`

{"services":[{"id":"d88856cb-fe3f-4473-ba8b-641480da810f","name":"memcached","description":"KubeDB managed Memcached","bindable":true,"plan_updateable":true,"plans":[{"id":"af1ce2dc-5734-4e41-aaa2-8aa6a58d688f","name":"demo-memcached","description":"Demo Memcached","free":true},{"id":"d40e49b2-f8fb-4d47-96d3-35089bd0942d","name":"memcached","description":"Memcached with custom specification","free":true}],"metadata":{"displayName":"KubeDB managed Memcached","imageUrl":"https://cdn.appscode.com/images/logo/databases/memcached.png"}},{"id":"938a70c5-f2bc-4658-82dd-566bed7797e9","name":"mysql","description":"KubeDB managed MySQL","bindable":true,"plan_updateable":true,"plans":[{"id":"1fd1abf1-e8e1-44a2-8214-bf0fd1ce9417","name":"demo-mysql","description":"Demo MySQL database","free":true},{"id":"6ed1ab9e-a640-4f26-9328-423b2e3816d7","name":"mysql","description":"MySQL database with custom specification","free":true}],"metadata":{"displayName":"KubeDB managed MySQL","imageUrl":"https://cdn.appscode.com/images/logo/databases/mysql.png"}},{"id":"2010d83f-d908-4d9f-879c-ce8f5f527f2a","name":"postgresql","description":"KubeDB managed PostgreSQL","bindable":true,"plan_updateable":true,"plans":[{"id":"c4bcf392-7ebb-4623-a79d-13d00d761d56","name":"demo-postgresql","description":"Demo Standalone PostgreSQL database","free":true},{"id":"41818203-0e2d-4d30-809f-a60c8c73dae8","name":"demo-ha-postgresql","description":"Demo HA PostgreSQL database","free":true},{"id":"13373a9b-d5f5-4d9a-88df-d696bbc19071","name":"postgresql","description":"PostgreSQL database with custom specification","free":true}],"metadata":{"displayName":"KubeDB managed PostgreSQL","imageUrl":"https://cdn.appscode.com/images/logo/databases/postgresql.png"}},{"id":"315fc21c-829e-4aa1-8c16-f7921c33550d","name":"elasticsearch","description":"KubeDB managed ElasticSearch","bindable":true,"plan_updateable":true,"plans":[{"id":"c4e99557-3a81-452e-b9cf-660f01c155c0","name":"demo-elasticsearch","description":"Demo Standalone Elasticsearch database","free":true},{"id":"2f05622b-724d-458f-abc8-f223b1afa0b9","name":"demo-elasticsearch-cluster","description":"Demo Elasticsearch cluster","free":true},{"id":"6fa212e2-e043-4ae9-91c2-8e5c4403d894","name":"elasticsearch","description":"Elasticsearch cluster with custom specification","free":true}],"metadata":{"displayName":"KubeDB managed ElasticSearch","imageUrl":"https://cdn.appscode.com/images/logo/databases/elasticsearch.png"}},{"id":"d690058d-666c-45d8-ba98-fcb9fb47742e","name":"mongodb","description":"KubeDB managed MongoDB","bindable":true,"plan_updateable":true,"plans":[{"id":"498c12a6-7a68-4983-807b-75737f99062a","name":"demo-mongodb","description":"Demo Standalone MongoDB database","free":true},{"id":"6af19c54-7757-42e5-bb74-b8350037c4a2","name":"demo-mongodb-cluster","description":"Demo MongoDB cluster","free":true},{"id":"e8f87ba6-0711-42db-a663-a3c75b78a541","name":"mongodb","description":"MongoDB database with custom specification","free":true}],"metadata":{"displayName":"KubeDB managed MongoDB","imageUrl":"https://cdn.appscode.com/images/logo/databases/mongodb.png"}},{"id":"ccfd1c81-e59f-4875-a39f-75ba55320ce0","name":"redis","description":"KubeDB managed Redis","bindable":true,"plan_updateable":true,"plans":[{"id":"4b6ad8a7-272e-4cfd-bb38-5b9d4bd3962f","name":"demo-redis","description":"Demo Redis","free":true},{"id":"45716530-cadb-4247-b06a-24a34200d734","name":"redis","description":"Redis with custom specification","free":true}],"metadata":{"displayName":"KubeDB managed Redis","imageUrl":"https://cdn.appscode.com/images/logo/databases/redis.png"}}]}

But seeing error while trying to provision a service:

curl -X PUT https://appscode-service-broker.appscode.svc/v2/service_instances/{instance_id} -H "Content-Type: application/json" -d '{"metadata":{"name":"demo-mysql","namespace":"appscode"},"spec":{"version":"8.0-v2","databaseSecret":{"secretName":"m1-auth"},"storageType":"Durable","storage":{"storageClassName":"gp2","accessModes":["ReadWriteOnce"],"resources":{"requests": {"storage":"1Gi"}}}}}' --resolve 'appscode-service-broker.appscode.svc:443:<IP>' --cacert cacert.crt

curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

and same error in pod logs as described in original comment:

http2: panic serving 192.168.244.88:57520: interface conversion: interface {} is nil, not string

Here's link to OSB API spec I am referring to.

bikramnehra avatar Feb 27 '19 19:02 bikramnehra

I have looked up the request format presented by service-catalog.

Code: https://github.com/kubernetes-incubator/service-catalog/blob/49e5cc4f90c6794d490eea4256061797223d9564/pkg/controller/controller_instance.go#L1635

Request Object Type Definition: https://github.com/kubernetes-incubator/service-catalog/blob/49e5cc4f90c6794d490eea4256061797223d9564/vendor/github.com/pmorie/go-open-service-broker-client/v2/types.go#L170

Code to generate OSB request from a ServiceInstance object: https://github.com/kubernetes-incubator/service-catalog/blob/master/pkg/controller/controller_instance.go#L2221

So, if you use the ServiceInstance object https://github.com/appscode/service-broker/blob/master/docs/examples/mysql-instance.yaml with service-catalog, it sends the following provision request to service-broker:

{
  "instance_id": "301c48b0-3e2e-11e9-b2eb-0242ac110006",
  "accepts_incomplete": true,
  "service_id": "938a70c5-f2bc-4658-82dd-566bed7797e9",
  "plan_id": "6ed1ab9e-a640-4f26-9328-423b2e3816d7",
  "organization_guid": "f5e1a041-3e2c-11e9-96f7-0242ac110007",
  "space_guid": "2f32b5ea-3e2e-11e9-af54-08002798a054",
  "parameters": {
    "metadata": {
      "labels": {
        "app": "my-mysql"
      }
    },
    "spec": {
      "storage": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "resources": {
          "requests": {
            "storage": "1Gi"
          }
        },
        "storageClassName": "standard"
      },
      "storageType": "Durable",
      "terminationPolicy": "WipeOut",
      "version": "8.0-v1"
    }
  },
  "context": {
    "clusterid": "f5e1a041-3e2c-11e9-96f7-0242ac110007",
    "namespace": "demo",
    "platform": "kubernetes"
  },
  "originatingIdentity": {
    "Platform": "kubernetes",
    "Value": "{\"username\":\"minikube-user\",\"uid\":\"\",\"groups\":[\"system:masters\",\"system:authenticated\"]}"
  }
}

@bikramnehra, I think the reason you are getting the error above is because you are just sending the parameter field as the request body, as I see from your curl command. If you send the full request object as shown above it should work.

You can also find the official spec for Provision request here: https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#provisioning

tamalsaha avatar Mar 04 '19 03:03 tamalsaha

I have created and pushed a new version of the service-broker that logs the request object as Docker image appscode/service-broker:reqlog. You can deploy or update your service-broker using the following command:

# install
$ helm install appscode/service-broker --name appscode-service-broker --namespace kube-system --set broker.tag=reqlog

# upgrade, if already installed
$ helm upgrade appscode-service-broker appscode/service-broker --namespace kube-system --set broker.tag=reqlog

Using this new image you can see actual request sent by service-catalog and replicate that with CF case.

tamalsaha avatar Mar 04 '19 03:03 tamalsaha 

{
  "instance_id": "97a3a018-e2d6-4a00-b5f2-6b894e5b74f9",
  "accepts_incomplete": true,
  "service_id": "938a70c5-f2bc-4658-82dd-566bed7797e9",
  "plan_id": "1fd1abf1-e8e1-44a2-8214-bf0fd1ce9417",
  "organization_guid": "d3cb4596-5d10-44e4-9998-c3419aa44963",
  "space_guid": "e2f4db2b-ba07-4ece-8993-bc0a2277913b",
  "parameters": {
    "metadata": {
      "labels": {
        "app": "my-mysql"
      }
    },
    "spec": {
      "storage": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "resources": {
          "requests": {
            "storage": "1Gi"
          }
        },
        "storageClassName": "standard"
      },
      "storageType": "Durable",
      "terminationPolicy": "WipeOut",
      "version": "8.0-v1"
    }
  },
  "context": {
    "organization_guid": "d3cb4596-5d10-44e4-9998-c3419aa44963",
    "platform": "cloudfoundry",
    "space_guid": "e2f4db2b-ba07-4ece-8993-bc0a2277913b"
  },
  "originatingIdentity": {
    "Platform": "cloudfoundry",
    "Value": "{\"user_id\":\"b73592f6-7ebf-4d6e-aa4b-963165209078\"}"
  }
}
{
  "instance_id": "97a3a018-e2d6-4a00-b5f2-6b894e5b74f9",
  "accepts_incomplete": true,
  "service_id": "938a70c5-f2bc-4658-82dd-566bed7797e9",
  "plan_id": "1fd1abf1-e8e1-44a2-8214-bf0fd1ce9417",
  "organization_guid": "d3cb4596-5d10-44e4-9998-c3419aa44963",
  "space_guid": "e2f4db2b-ba07-4ece-8993-bc0a2277913b",
  "parameters": {
    "metadata": {
      "labels": {
        "app": "my-mysql"
      }
    },
    "spec": {
      "storage": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "resources": {
          "requests": {
            "storage": "1Gi"
          }
        },
        "storageClassName": "standard"
      },
      "storageType": "Durable",
      "terminationPolicy": "WipeOut",
      "version": "8.0-v1"
    }
  },
  "context": {
    "organization_guid": "d3cb4596-5d10-44e4-9998-c3419aa44963",
    "platform": "cloudfoundry",
    "space_guid": "e2f4db2b-ba07-4ece-8993-bc0a2277913b"
  },
  "originatingIdentity": {
    "Platform": "cloudfoundry",
    "Value": "{\"user_id\":\"b73592f6-7ebf-4d6e-aa4b-963165209078\"}"
  }
}

E0304 23:48:26.890627       1 runtime.go:67] Observed a panic: interface conversion: interface {} is nil, not string
goroutine 4244 [running]:
github.com/appscode/service-broker/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc0009a6240)
	/go/src/github.com/appscode/service-broker/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:103 +0xe1
panic(0x1a21320, 0xc000279f50)
	/usr/local/go/src/runtime/panic.go:513 +0x1b9

bikramnehra avatar Mar 04 '19 23:03 bikramnehra

I0305 00:04:29.469603 1 apisurface.go:209] Unable to retrieve originating identity - unable to find originating identity

{
  "instance_id": "1b0538e0-c8d8-43be-8362-2f44ff72bf5a",
  "accepts_incomplete": true,
  "service_id": "938a70c5-f2bc-4658-82dd-566bed7797e9",
  "plan_id": "1fd1abf1-e8e1-44a2-8214-bf0fd1ce9417"
}
I0305 00:04:29.469721       1 broker.go:124] Deprovisioning instance "1b0538e0-c8d8-43be-8362-2f44ff72bf5a" for "938a70c5-f2bc-4658-82dd-566bed7797e9"/"1fd1abf1-e8e1-44a2-8214-bf0fd1ce9417"...
{
  "instance_id": "ec7c1aae-6f68-4194-80c2-49b1ca47069b",
  "accepts_incomplete": true,
  "service_id": "938a70c5-f2bc-4658-82dd-566bed7797e9",
  "plan_id": "1fd1abf1-e8e1-44a2-8214-bf0fd1ce9417",
  "organization_guid": "d3cb4596-5d10-44e4-9998-c3419aa44963",
  "space_guid": "e2f4db2b-ba07-4ece-8993-bc0a2277913b",
  "parameters": {
    "metadata": {
      "labels": {
        "app": "my-mysql"
      }
    },
    "spec": {
      "storage": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "resources": {
          "requests": {
            "storage": "1Gi"
          }
        },
        "storageClassName": "standard"
      },
      "storageType": "Durable",
      "terminationPolicy": "WipeOut",
      "version": "8.0-v1"
    }
  },
  "context": {
    "organization_guid": "d3cb4596-5d10-44e4-9998-c3419aa44963",
    "platform": "cloudfoundry",
    "space_guid": "e2f4db2b-ba07-4ece-8993-bc0a2277913b"
  },
  "originatingIdentity": {
    "Platform": "cloudfoundry",
    "Value": "{\"user_id\":\"b73592f6-7ebf-4d6e-aa4b-963165209078\"}"
  }
}

bikramnehra avatar Mar 05 '19 00:03 bikramnehra

appscode_pod.log

bikramnehra avatar Mar 05 '19 00:03 bikramnehra 

{
  "metadata": {
    "labels": {
      "app": "my-mysql"
    }
  },
  "spec": {
    "storage": {
      "accessModes": [
        "ReadWriteOnce"
      ],
      "resources": {
        "requests": {
          "storage": "1Gi"
        }
      },
      "storageClassName": "gp2"
    },
    "storageType": "Durable",
    "terminationPolicy": "WipeOut",
    "version": "8.0-v1"
  }
}

$ cf -vvv create-service mysql mysql mysql-example-svc16 -c /home/tamal/Desktop/my.json

Issues:

  • [x] namespace was expected in the request.context
    • https://github.com/appscode/service-broker/commit/74ebc994f9ccb4c7aee7228f18e468b79420c4ff#diff-55aca6fb74ac953881309c295ee9441eL53
  • [x] Kubedb operator was not installed
  • [x] DB crd object name was extracted from ServiceInstance name. With CF, there is no ServiceInstance. So, instance name has to be generated some other way.
    • https://github.com/appscode/service-broker/commit/c9ca9fd4f1d54a035c1c53efa8a298cefa0d384a#diff-55aca6fb74ac953881309c295ee9441eL73

Follow-ups

  • How to determine name and namespace of the DB CRD object (instance) in CF setup?

tamalsaha avatar Mar 05 '19 01:03 tamalsaha

For Namespace:

Here's how its handled in minibroker:

https://github.com/osbkit/minibroker/commit/c93033f5386e2a2f1f439da9363db8160688db64#diff-55aca6fb74ac953881309c295ee9441e

In case the namespace is provided then use it otherwise use the default namespace(which is most likely the namespace where the broker itself is provisioned).

For DB CRD naming:

This can be handled in multiple ways, though I strongly feel it must be abstracted from the user. Initially I was thinking about using the SERVICE_INSTANCE name from the cf create-service request, however the broker won't see this value as cf will pass the identifier in the request and not the name.

cf create-service SERVICE PLAN SERVICE_INSTANCE [-c PARAMETERS_AS_JSON]

So I guess may be we can agree on some convetion to ensure uniqueness and keep it consistent throughout. e.g:

<DB_CRD_OBJ>-<InstanceID> mysql-1b0538e0-c8d8-43be-8362-2f44ff72bf5a

bikramnehra avatar Mar 07 '19 23:03 bikramnehra

Getting the following error while trying to create-service:

$ cf create-service mysql mysql mysql-example-svc
Creating service instance mysql-example-svc in org org / space space as admin...
FAILED
Server error, status code: 502, error code: 10001, message: Service broker error: the server could not find the requested resource (get mysqls.kubedb.com)

and seeing following error messages in the pod logs:

I0327 18:47:24.931966       1 apisurface.go:207] Unable to retrieve originating identity - unable to find originating identity
I0327 18:47:24.931986       1 broker.go:133] Deprovisioning instance "25ef5ae0-63f7-4199-9514-5aa2e211b90d" for "938a70c5-f2bc-4658-82dd-566bed7797e9"/"1fd1abf1-e8e1-44a2-8214-bf0fd1ce9417".

bikramnehra avatar Mar 27 '19 18:03 bikramnehra

the server could not find the requested resource (get mysqls.kubedb.com)

Sounds like you have not installed the KubeDB operator. https://github.com/kubedb/cli/blob/0.11.0/docs/setup/install.md#helm

tamalsaha avatar Mar 27 '19 18:03 tamalsaha

Thanks @tamalsaha for pointing that out, I installed the KubeDB operator and re-triggered the create-service command and now seeing following error message:

$ cf create-service mysql mysql mysql-example-svc
Creating service instance mysql-example-svc in org org / space space as admin...
FAILED
Server error, status code: 502, error code: 10001, message: Service broker error: failed to create 938a70c5-f2bc-4658-82dd-566bed7797e9 obj "app-b587b7a8-b4b6-46bf-899d-cdc4bb91c410" in namespace default: spec is required for provisioning custom postgres

not sure why am I seeing postgres related errors while I am trying to provision mysql

bikramnehra avatar Mar 27 '19 20:03 bikramnehra

If you do cf create-service mysql demo-mysql mysql-example-svc, it should work without any additional configuration. For mysql plan, you need provide a spec via parameters

      spec:
        storage:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi
          storageClassName: standard
        storageType: Durable
        terminationPolicy: WipeOut
        version: 8.0-v1

tamalsaha avatar Mar 27 '19 21:03 tamalsaha

I see, also tried my demo-mysql, here's the output:

$ cf create-service mysql demo-mysql mysql-example-svc
Creating service instance mysql-example-svc in org org / space space as admin...
FAILED
Server error, status code: 502, error code: 10001, message: Service broker error: failed to create 938a70c5-f2bc-4658-82dd-566bed7797e9 obj "app-aeef8e53-ae91-4623-a618-3b0c42f355d0" in namespace default: admission webhook "mysql.validators.kubedb.com" denied the request: mysqlversions.catalog.kubedb.com "8.0-v1" not found

bikramnehra avatar Mar 27 '19 22:03 bikramnehra

I think you did not fully install the KubeDB operator. This is missing the KubeDB catalog.

Step 3(a): Install KubeDB catalog of database versions

$ helm install appscode/kubedb-catalog --name kubedb-catalog --version 0.11.0
--namespace kube-system

tamalsaha avatar Mar 27 '19 22:03 tamalsaha

After following your comments I am able to get create-service up and running. I can also see the pods spawning in the default namespace which was the expected behavior. Thanks, @tamalsaha commendable work.

However, I am currently seeing two issues in testing the app binding part:

  1. Currently, I am only able to provision the default plans without any custom parameters, which I guess will restrict testing with any custom apps.

  2. Currently, for testing minibroker we are relying on the following apps:

https://github.com/scf-samples/cf-redis-example-app/tree/42ac509e68bf4aec8ba1bb998c558315e2dea4cf

https://github.com/scf-samples/rails-example/tree/fc79c69df8149cb8debb83a945791a2529a0ce35

Some of these apps need might need some custom settings in the service in order for them to be bindable to that service, we might want to check and possibly have those settings in the operator.

bikramnehra avatar Mar 28 '19 01:03 bikramnehra

Binding

I did perform some testing with mongodbblog sample app. Here are my observations:

I was able to successfully create the app and bind mongodb service instance to the app, however, I started seeing issues as soon as I started posting data to the app. See test for more details.

$ curl -L -v --fail -X POST http://mongoblog.cap.bsingh.aws.howdoi.website/post/new --data 'post[title]=desired-title' --data 'post[body]=desired-body'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 52.32.185.182...
* TCP_NODELAY set
* Connected to mongoblog.cap.bsingh.aws.howdoi.website (52.32.185.182) port 80 (#0)
> POST /post/new HTTP/1.1
> Host: mongoblog.cap.bsingh.aws.howdoi.website
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Length: 49
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 49 out of 49 bytes
* The requested URL returned error: 500 Internal Server Error
* stopped the pause stream!
* Closing connection 0
curl: (22) The requested URL returned error: 500 Internal Server Error

I also noticed that there were no pods spawned while creating mongodb service instance which might be a probable cause for the problem.

Service Catalog

Also, in regards to the issue, I mentioned regarding service-catalog as soon as I deleted the service-catalog I started seeing errors in the broker. Here is an example:

$ cf create-service postgresql demo-postgresql postgresql-example-svc
Creating service instance postgresql-example-svc in org org / space space as admin...
Service broker error: the server could not find the requested resource (get serviceinstances.servicecatalog.k8s.io)
FAILED

Here' s the list of resources currently installed on the cluster:

$ helm list
NAME                   	REVISION	UPDATED                 	STATUS  	CHART                	APP VERSION	NAMESPACE
appscode-service-broker	1       	Wed Mar 27 10:51:46 2019	DEPLOYED	service-broker-0.2.0 	0.2.0      	appscode
kubedb-catalog         	1       	Wed Mar 27 16:52:38 2019	DEPLOYED	kubedb-catalog-0.11.0	0.11.0     	appscode
kubedb-operator        	1       	Wed Mar 27 16:54:21 2019	DEPLOYED	kubedb-0.11.0        	0.11.0     	appscode
scf                    	1       	Tue Mar 26 11:55:09 2019	DEPLOYED	cf-2.15.2            	1.3.1      	scf
uaa                    	1       	Tue Mar 26 11:41:20 2019	DEPLOYED	uaa-2.15.2           	1.3.1      	uaa

bikramnehra avatar Mar 28 '19 06:03 bikramnehra

@bikramnehra , I have cut 0.3.0 release of this service broker project. It should fix both the issues you mentioned above. Also, I have added a new values field catalog.controller.enabled. https://github.com/appscode/service-broker/blob/master/chart/service-broker/values.yaml#L104

You can use this to disable any service catalog specific parts from the service broker chart.

helm install appscode/service-broker --name appscode-service-broker \
  --namespace kube-system \
  --set catalog.controller.enabled=false

tamalsaha avatar Mar 28 '19 11:03 tamalsaha

I tested again with 0.3.0 and now I can see the pod spawning for the service instance. Also, now I am able to work with the broker without Service Catalog. However, there seem to be some issues as I am trying to post something to DB:

curl -L -v --fail -X POST http://mongoblog.cap.bsingh.aws.howdoi.website/post/new --data 'post[title]=desired-title' --data 'post[body]=desired-body'

It seems like authentication related failures:

2019-03-28T17:37:48.938+0000 I NETWORK  [conn257] end connection 127.0.0.1:49206 (0 connections now open)
2019-03-28T17:37:49.115+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:49208 #258 (1 connection now open)
2019-03-28T17:37:49.116+0000 I NETWORK  [conn258] received client metadata from 127.0.0.1:49208 conn258: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.6.11" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 9 (stretch)"", architecture: "x86_64", version: "Kernel 4.14.88-88.76.amzn2.x86_64" } }
2019-03-28T17:37:49.119+0000 I ACCESS   [conn258] Unauthorized: not authorized on admin to execute command { endSessions: [ { id: UUID("53ea865d-9e3f-4de2-9f05-4817bd1bae67") } ], $db: "admin" }

Now the question is how do we authenticate the requests coming via CloudFoundry client.

bikramnehra avatar Mar 28 '19 17:03 bikramnehra

Also, tried it for the redis app still seem to be getting similar issues:

* Connected to redis-example-app.cap.bsingh.aws.howdoi.website (52.32.185.182) port 80 (#0)
> PUT /hello HTTP/1.1
> Host: redis-example-app.cap.bsingh.aws.howdoi.website
> User-Agent: curl/7.64.0
> Accept: */*
> Content-Length: 12
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 12 out of 12 bytes
< HTTP/1.1 500 Internal Server Error
< Content-Length: 238
< Content-Type: text/html;charset=utf-8
< Date: Thu, 28 Mar 2019 18:33:55 GMT
< Server: WEBrick/1.3.1 (Ruby/2.4.5/2018-10-18)
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Vcap-Request-Id: f8670d2b-d678-489c-5116-8ec3b07f57bd
< X-Xss-Protection: 1; mode=block
<

Doesn't seem to find much in the pod logs in this case, and the error response 500 is vague to identify any possible issues.

That makes me wonder is it has anything to do with the fact that I am using demo-plans for provisioning these services? Though clearly, the app is able to bind successfully I am not seeing anything populated in the VCAP_SERVICES variable:

{
 "VCAP_SERVICES": {
  "redis": [
   {
    "binding_name": null,
    "credentials": {
     "Host": "app-295f0237-a80d-4470-8b66-711280433c6a.default.svc",
     "Password": null,
     "Port": 6379,
     "Protocol": "redis",
     "RootCert": null,
     "URI": "redis://app-295f0237-a80d-4470-8b66-711280433c6a.default.svc:6379",
     "Username": null
    },
    "instance_name": "redis-example-svc",
    "label": "redis",
    "name": "redis-example-svc",
    "plan": "demo-redis",
    "provider": null,
    "syslog_drain_url": null,
    "tags": [],
    "volume_mounts": []
   }
  ]
 }
}

You can refer this link to see how credentials are bound to apps.

bikramnehra avatar Mar 28 '19 18:03 bikramnehra

I tested again with 0.3.0 and now I can see the pod spawning for the service instance. Also, now I am able to work with the broker without Service Catalog.

Great!

However, there seem to be some issues as I am trying to post something to DB:

curl -L -v --fail -X POST http://mongoblog.cap.bsingh.aws.howdoi.website/post/new --data 'post[title]=desired-title' --data 'post[body]=desired-body'

It seems like authentication related failures:

We have configured auth for databases by default. So, you need to provide auth to connect. Bind operator is supposed to bind credentials for apps.

Now the question is how do we authenticate the requests coming via CloudFoundry client.

What command is CloudFoundry client is running? Why is this trying to access database?

tamalsaha avatar Mar 28 '19 20:03 tamalsaha

I should have been more clear while describing the issue, the cf client is not trying to connect to the database, but the app that is deployed. So the curl request you see above is suppose to create an entry in the database which we usually assert to:

https://github.com/SUSE/scf/blob/develop/src/scf-release/src/acceptance-tests-brain/test-scripts/016_minibroker_mongodb_test.rb#L65-L70

As mentioned above, I am seeing 500 Internal Server Error while doing this and not much info from the pod logs to be absolutely sure about the exact issue.

bikramnehra avatar Mar 28 '19 20:03 bikramnehra