Suggestion: Option to select connectors to load

[simplesteph]

Is it possible to have an environment variable specifying the connectors we're interested into loading? By default all would be loaded

The goal is to have a lighter development environment and load exactly the connectors we'd like to try.

[andmarios]

Hello! For now we have the option to disable connectors but your suggestion is good. I will explore it and come back to you.

[andmarios]

Just pushed 62dd7786208e122b7ed65ee0da8acd8464fa853c which adds the CONNECTORS env var in order to explicitly enable connectors. So now you can do:

docker run -e CONNECTORS=jdbc,elastic landoop/fast-data-dev

If you don't enable the jdbc connector, some connect tests will fail but it doesn't otherwise affect the operation of fast-data-dev.

Currently this is only on the master branch (latest tag). Once tested a bit more, I will merge it to the cp4.0 branch/tag which has Kafka 1.0. :)

[simplesteph]

Awesome :) Can we get a list of connectors names and connectors abbreviation documented for the CONNECTORS= option? Additionally, for elastic, is there a way to specify the confluent one? Cheers! Stephane

[andmarios]

Yes, we should add a list, will do! For now it is the part of the directory name after kafka-connect-

To see them all you can run:

docker run --rm -it landoop/fast-data-dev \
    find /opt/confluent/share/java /opt/connectors -type d -maxdepth 1 -name "kafka-connect-*"

Confluent's ES connector is named elasticsearch. Landoop's are elastic and elastic5.

[andmarios]

Hi @simplesteph, a small heads-up.

Have a look at landoop/fast-data-dev:1.0.1. This is the next generation fast-data-dev and the reason it took us so long to prepare it.

It may look similar to the old one but it ain't. It starts faster, it is more robust, it uses less RAM. These are some of the immediate changes you may notice.

All components are build by us and Kafka now comes from the official Apache Kafka repository.

Also many changes happened in the background and now we support configuring Kafka and the rest of the services. This means we can even support chaining many fast-data-dev instances to create a proper cluster!

The goodies don't stop there though. Set BROWSECONFIGS=1 and you will be able to see the configuration files of all running components. Set SUPERVISORWEB=1 and you will get the supervisor control interface on port 9001. There you even have a tail -f option for the logs.

[simplesteph]

@andmarios thanks, that's very exciting! I'll have a look :)