Confusing notation of spring.cloud.deployer.* and spring.cloud.task.platform.* properties

[sayapeg]

Thank you for all the work on SCDF!!

Problem Description: The SCDF docs specify various ways to set properties on the application and how they get deployed on various platforms. It is confusing for a new user to immediately grasp these notations, specifically between the deployer and task properties. For example,

For example, under 11.2. Application and Server Properties, it says how to customize the application deployment behaviour - CPU, memory, K8S settings, etc. In the next section under 11.2.1. Memory and CPU Settings, it shows examples for deployer.<application>.kubernetes.limits.cpu=1000m, spring.cloud.skipper.server, and spring.cloud.dataflow.task. In the next section 11.2.2. Environment Variables, it says spring.cloud.deployer.kubernetes.environmentVariables can be used to set environment variables, but the example says deployer.<application>.kubernetes.environmentVariables.

Solution Description: Clarify if these are correct:

• None of the deployer.*, spring.cloud.deployer.*, spring.cloud.dataflow.task.platform, and spring.cloud.skipper.server.platform properties make sense in the application. They are all for configuring how the application is deployed in local/k8s/cf environments.
• deployer.* is a shortcut for spring.cloud.deployer.*. In turn, this means spring.cloud.deployer.<application>.<propertyName> and spring.cloud.deployer.<propertyName> are equivalent for application-specific and global properties. Another way is deployer.* should be when launching the application from UI/shell, and spring.cloud.deployer.* are configured in the SCDF Server/Skipper.
• spring.cloud.deployer.* is per platform (k8s) and application deployed on that platform. spring.cloud.skipper.server.*, and spring.cloud.dataflow.task.* is for platform accounts and streams/tasks properties.

Considering the above, and the below scenario, running the hello-world in the default account makes CPU requests 1750, and in the dev account, it would be 1500. And if I don't provide a CPU request for an account named qa, would it be 1200 for hello-world?

spring.cloud.deployer.kubernetes.requests.cpu=1000
spring.cloud.deployer.hello-world.kubernetes.requests.cpu=1200
spring.cloud.dataflow.task.platform.kubernetes.default.requests.cpu=1750
spring.cloud.dataflow.task.platform.kubernetes.dev.requests.cpu=1500

Description of alternatives: Clarify in the docs of any shortcuts or abbreviations.

Additional context: Easier understanding for new users.

[sayapeg]

Just now, I tried spring.cloud.deployer.kubernetes.createJob=true for the SCDF server and deployer.hello-world.kubernetes.createJob=true to launch the task as a job, and both of them didn't work. So I am not sure anymore.

[cppwfs]

• You are correct. Deployment properties are only for configuring how an app is deployed and not the application properties.
• deployer.*.<myprop> applies the <myprop> deployer property to all the apps comprising a composed task. (if we are discussing tasks) .
• I'm not sure we have global properties for multiple platforms. But I could be wrong @onobc or @corneil feel free to correct me.
• For the question pertaining to the value that will be applied for requests.cpu it would be 1200, because we are directing SCDF to apply the value to a specific app deployment.

For your last question currently, SCDF/Kubernetes Deployer only supports setting the create-job as a SCDF default and not per launch. For example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: scdf-server
  labels:
    app: scdf-server
data:
  application-kubernetes.yaml: |-
    logging:
      level:
        root: info
        org.springframework: debug
    management:
      defaults:
        metrics:
          export:
            enabled: false
    spring:
      output:
        ansi:
          enabled: NEVER
      cloud:
        deployer:
          kubernetes:
            imagePullPolicy: IfNotPresent
            imagePullSecret: registry-key
        dataflow:
          metrics.dashboard:
            url: 'http://localhost:3000'
          task:
            platform:
              kubernetes:
                accounts:
                  default:
                    imagePullPolicy: IfNotPresent
                    imagePullSecret: registry-key
                    limits:
                      memory: 1024Mi
                    createJob: true

However, I have created an issue on the Deployer project to discuss this: https://github.com/spring-cloud/spring-cloud-deployer/issues/397

[sayapeg]

Thank you, @cppwfs, for your response. I still couldn't understand the difference between the deployer.* and the spring.cloud.deployer.* properties. Is this the difference? deployer.* should be when launching the application from UI/shell, and spring.cloud.deployer.* are configured in the SCDF Server/Skipper.

And regarding the CPU value, if I specify both spring.cloud.deployer.hello-world.kubernetes.requests.cpu=1200 and spring.cloud.dataflow.task.platform.kubernetes.default.requests.cpu=1750, launching the task in default platform account, why the CPU would be 1200 instead of 1750? spring.cloud.deployer has higher priority than spring.cloud.dataflow.task.platform?

[cppwfs]

For the first question: Spring Cloud Data Flow uses the Spring Cloud Deployer to deploy its task and stream applications to the platform. Spring Cloud Deployer uses the spring.cloud.deployer prefix for its properties so users can configure how to deploy apps to the platform. SCDF abbreviates this to deployer.<name of app>.property=value i.e. deployer.hello-world.kubernetes.requests.cpu=XXXX. So when configuring the deployment properties for an application using SCDF use the format deployer.<name of app>.property=value .

For the second question: Yes.

[sayapeg]

I got it. Thank you for the clarifications. Can we have a naming convention section explaining these things? I think the information of SCDF using the spring cloud deployer project is already in the docs.