Automate bootstrap for non-containerized deployment?

[orpiske]

Looking at the AWS-to-JMS example it seems to me that the bootstrap steps are going to be similar most of the time.

1. Download the connector package
2. (Optional) Download the dependencies
3. Create the configuration file
4. Put the files on the Kafka Connect runtime install directory.

I know this is handled automatically that for containerized deployments such as Strimzi, so I was wondering if the project should have a facility for regular deployments.

What do you think? IMHO, it could make even the documentation and example creation simpler, as the project can abstract a few repetitive things and reduce the entry barrier.

[valdar]

It sounds a good idea to me! In which form would you plan to add that?

[orpiske]

It sounds a good idea to me! In which form would you plan to add that?

Thanks.

I was thinking about a CLI utility, in a module within the project. The usage would look like this:

camel-kafka-connector bootstrap --connector sjms2 --target-dir /path/to/kafka/connect

It would have 2 additional arguments:

--download-host: to allow choosing the host from where to download the files
--override-property: to allow the user to override a setting

We already have the list of configurations in the connector config files (example).

It's not clear to me yet how to handle generating the configuration. I think there are 2 options:

1. We could simply generate a configuration file with all the settings.
2. We could generate a configuration file w/ only the required settings.

Option 1 is significantly easier, although the configuration file would be much larger than our examples. Option 2 is more complicated and we would probably need to come up with a mechanism to highlight what is required and what is not.

IMHO, I think we could stick w/ option 1.

[lburgazzoli]

looks looks we may need a start.spring.io alike solution but for camel

[valdar]

@orpiske For the configuration part actually I was planning to add an autogenerated example configuration listing only the required options (the camel catalog should provide that information). We can potentially convey that information using the Kafka connect config priority field (i.e. only the properties with priority = HIGH are the mandatory ones).

Probably we would need to specify the version as well? Or directly some GAV?

It is a nice idea, it sounds a decent amount of work is needed for a facility to basically run examples and standalone (that nowadays I am not sure how much would be used, as you said containers platforms have these facilities built in somehow.

@lburgazzoli well actually this would be something different since it does not generate a starting project for you but rather help you to easily run one. Have I missed something?

[lburgazzoli]

@lburgazzoli well actually this would be something different since it does not generate a starting project for you but rather help you to easily run one. Have I missed something?

I think I misinterpreted the subject reading the issue on my mobile as points 1,2,3 looks very similar to what you do with start.spring.io or similar tools.

[orpiske]

@orpiske For the configuration part actually I was planning to add an autogenerated example configuration listing only the required options (the camel catalog should provide that information). We can potentially convey that information using the Kafka connect config priority field (i.e. only the properties with priority = HIGH are the mandatory ones).

I think that's a good idea. I think reusing the priority would make it possible to use it for the bootstrapping part as well.

Probably we would need to specify the version as well? Or directly some GAV?

I think we need that too. I think we could use the GAV. IIRC, camel-k CLI already uses the GAV, so overall usage throughout the projects would be consistent.

It is a nice idea, it sounds a decent amount of work is needed for a facility to basically run examples and standalone (that nowadays I am not sure how much would be used, as you said containers platforms have these facilities built in somehow.