The Rest API does not allow you to specify deployment variables

[Flooze-IT]

Hi everyone,

Description: I am currently working with SCDF since a couple of month and i particulary use the REST API. To bring some context, I use scdf as part of Kubernetes.

My problem is the following: I need to specify deployment variables from the API. However when in my DSL I specify "--deployer....", SCDF overrides this information by prefixing with "app.<nameOfMyApp>.deployer...". Is this a bug or a misunderstanding on my part?

I need it to specify the name of the secret to fetch the docker image or add annotations to my POD.

Thank you in advance for your help

Release versions: docker.io/bitnami/spring-cloud-dataflow:2.9.4-debian-11-r3

Steps to reproduce: From REST Api myapp --deployer.stream-connector-nexus.kubernetes.image-pull-secrets=regcred

override by

myapp --app.myapp.deployer.stream-connector-nexus.kubernetes.image-pull-secrets=regcred

[corneil]

Can you provide a simple flow definition and a sample curl that you execute.

Also take note that deployer properties can be added to the values.yml when using the Helm chart.

Skipper Getting Started

[Flooze-IT]

Thanks @corneil

Complete URL: http://localhost:8080/streams/definitions?name=pl52-app-test-mon-topic-saas&definition=stream-connector-nexus+--deployer.stream-connector-nexus.kubernetes.image-pull-secret%3Dregcred+--deployer.stream-connector-nexus.kubernetes.image-pull-secrets%3Dregcred+--spring.config.import%3Doptional%3Aconfigserver%3Ahttp%3A%2F%2Fconfig-server-service.default.svc.cluster.local%3A8888+--spring.application.name%3Dpl52-app-test-mon-topic-saas&description=Semi+conector+for+PL52+related+to+app%3A+APP_TEST+on+Topic+%3A+com.bl.blexchange.kafkaserviceconfiguration.models.InputTopic%407ab04d22+with+a+deployment%3A+SAAS&deploy=true

Queries: name: pl52-app-test-mon-topic-saas definition: stream-connector-nexus+--deployer.stream-connector-nexus.kubernetes.image-pull-secret%3Dregcred+--deployer.stream-connector-nexus.kubernetes.image-pull-secrets%3Dregcred+--spring.config.import%3Doptional%3Aconfigserver%3Ahttp%3A%2F%2Fconfig-server-service.default.svc.cluster.local%3A8888+--spring.application.name%3Dpl52-app-test-mon-topic-saas description: Semi+conector+for+PL52+related+to+app%3A+APP_TEST+on+Topic+%3A+com.bl.blexchange.kafkaserviceconfiguration.models.InputTopic%407ab04d22+with+a+deployment%3A+SAAS deploy: true

We want to add deployer properties dynamically when we request the SCDF Rest API

[Flooze-IT]

up ! Someone have any ideas ?

[markpollack]

This could be a bug. Classifying it as such and scheduling for the next release.

[onobc]

Hi @Flooze-IT

• The stream definition DSL considers any properties as application properties, not deployment properties.
• The "deploy" API supports deployment properties by setting them as the request body on the POST request.
• The "create-with-deploy" API does not support deployment properties as request body on the POST request.

Workaround

You can get around this limitation by doing a "create-without-deploy" followed by a "deploy" such as:

curl -X POST 'http://localhost:9393/streams/definitions?name=foostream4&definition=time%20%7C%20log&description=foostream4&deploy=false'

curl 'http://localhost:9393/streams/deployments/foostream4' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  --data-raw '{"deployer.*.foo4":"bar4"}' \
  --compressed

Enhancement

I am planning on adding support to the "create-with-deploy" API to accept a request body containing the deployment properties.

[Flooze-IT]

Many thanks @onobc !