Common properties for child Task in SCDF Composed-Task

[szopal]

I can see problem with pass common parameters like data source parameter to all child Task. For example if I have 2 task (Task1 and Task2) I have to pass arguments and parameters on this way:

--increment-instance-enabled=true

--interval-time-between-checks=1000

--dataflow-server-uri=http://localhost:9015

--spring.datasource.driverClassName=org.postgresql.Driver

--spring.datasource.password=xxx

--spring.datasource.url=jdbc:postgresql://xxxxx

--spring.datasource.username=xxx

--spring.datasource.hikari.schema=xxx

--composed-task-arguments="parm1=some_value"

app.basictransition.task1.spring.datasource.driverClassName=org.postgresql.Driver app.basictransition.task1.spring.datasource.password=xxx app.basictransition.task1.spring.datasource.url=jdbc:postgresql://xxxxx app.basictransition.task1.spring.datasource.username=xxx app.basictransition.task1.spring.datasource.hikari.schema=xxx app.basictransition.task1.increment-instance-enabled=true app.basictransition.task2.spring.datasource.driverClassName=org.postgresql.Driver app.basictransition.task2.spring.datasource.password=xxxx app.basictransition.task2.spring.datasource.url=jdbc:postgresql://xxxxx app.basictransition.task2.spring.datasource.username=xxx app.basictransition.task2.spring.datasource.hikari.schema=xxx app.basictransition.task2.increment-instance-enabled=true

It would be nice to pass common datasource parameters to all task included in Composed Task.

[cppwfs]

Hello @szopal , Just wanted to clarify your request. By default SCDF passes the spring.datasource.* property information to the composed task runner. Each task that is launched by the composed task runner will also receive the spring.datasource.* . So you do not have to establish the datasource properties for each task in the graph (unless you are using multiple databases in each of your tasks, but that is not shown in the example above.). If you are seeing a different behavior, please let us know.

Now as to other properties we could add a feature that would allow you to specify a wild card for the task name. i.e. app.basictransition.*.increment-instance-enabled=true .

[djgeary]

I think this is the same as this issue that was raised against the composed task runner https://github.com/spring-cloud-task-app-starters/composed-task-runner/issues/13

We would like this functionality too - examples would be common properties such as urls or external databases (ie other than the spring.datasource.* one) as well as common deployment properties.

For example if you had a simple task that executes a piece of SQL (eg https://github.com/djgeary/sql-executing-task/tree/master/src/main/java/com/example/task) against an external database then we could chain several of these together using a composed task and only have to set common properties for the target database instead of setting the same ones on each sub task.

The wild card format would work well although we would probably also want an explicit one to override the wildcard eg setting common kubernetes deployment properties for all sub tasks except one that requires more memory.

[sabbyanandan]

@cppwfs: Any other (latest) thoughts to share on this subject?

[cppwfs]

@sabbyanandan No other comment, other than we probably can add this as a feature to CTR.