Add Kafka brokers list support as environment variable

[Constantin07]

It would be nice to support supplying the list of Kafka broker for the provider via environment variable so that the config file doesn't need to be modified for different environments.

PS: In similar way as KAFKA_CA_CERT, something like BOOTSTRAP_SERVERS ...

BOOTSTRAP_SERVERS=broker1:9094,broker2:9094,broker3:9094

[OneCricketeer]

Can you not use TF_VAR_name?

[Constantin07]

Giving some context: I'm using AWS MSK. It does return the broker list as:

broker1:9094,broker2:9094,broker3:9094

Using TF_VAR_name would require mangling this connection string to transform it into:

TF_VAR_name='[broker1:9094, broker2:9094, broker3:9094]'

which in my opinion is just unnecessary over complication. For instance, Kafka command line utilities do accept broker1:9094,broker2:9094,broker3:9094 format.

Looking at provider.go other parameters already do support environment variables.

It would be nice to have env. var support for this too, isn't it? It's just makes easier from configuration point of view - no need for changes in files, no need to mangle TF_VARs.

[OneCricketeer]

I'm not sure I understand why you would need to set the variable to a list rather than a comma separated string

[Constantin07]

@cricket007 probably I wasn't clear enough. Terraform plan for MSK cluster returns:

Outputs:

bootstrap_brokers_tls = b-2.infratest-test-msk.a3g3juk.c3.kafka.eu-west-1.amazonaws.com:9094,b-3.infratest-test-msk.a3g3juk.c3.kafka.eu-west-1.amazonaws.com:9094,b-1.infratest-test-msk.a3g3juk.c3.kafka.eu-west-1.amazonaws.com:9094

That's how AWS TF provider returns the brokers.

I would like to pass bootstrap_brokers_tls as it is as env. variable to another TF plan to create topics, acls, without any changes.

At the moment this provider config accepts bootstrap_servers as a list not as comma separated sting ...

Does it make sense now ?

[OneCricketeer]

I do understand, thanks. However, that seems to be a feature unrelated to this provider, then.

Specifically, I still don't see what's stopping you from doing terraform plan msk.tf > file.details > (parse file) > export TF_VAR_bootstrap=mskAddress

e.g

terraform plan | grep brokers | cut -d= -f2 | tr -d [:space:]

[Constantin07]

@cricket007 why it is a feature unrelated to this provider ?

All other options of this provider do support env. variables. Just being consistent to have support for BOOTSTRAP_SERVERS will make it better and easier to use.

For instance, Kafka command line utilities also expect brokers in this format. What's wrong with making it a little bit more friendly ?

[OneCricketeer]

What's wrong with making it a little bit more friendly

Nothing.

I'm saying that the output of your external scripts are unrelated to any issues here.

And there's valid workarounds here, as discussed

Your ultimate goal is to not have separate env scripts, and that's already possible

[ryanhall07]

Does TF_VAR_name work since you can't use variables in providers?

https://github.com/hashicorp/terraform/issues/11578

[OneCricketeer]

I'm sure there is a workaround. I don't know it at the moment.

[gracemukendi]

Hi @OneCricketeer

Reading the comments above, I also encountered this issue today. I have a similar situation in that I am creating my MSK and my kafka topics within the same terraform module, and what would be the natural thing to do would be to pass the resulting msk's broker list (after creation) to my kafka provider (which seems to take vars) and have my topics created.

But I can't do that because the list of brokers is returned as env vars rather than a list of strings.

provider "kafka" { bootstrap_servers = aws_msk_cluster.demo.bootstrap_brokers }

It would seem to be quite a useful feature to have the kafka provider support having the list of brokers injected as env vars.

Is this something that would be worth looking into?

Thanks.

[OneCricketeer]

support supplying the list of Kafka broker for the provider via environment variable

This is all I was responding too. It already is possible via TF_VAR support, which is applicable to all variables.

If you need to "mangle" the string together using external tooling than Terraform, then that is a separate issue

[danielmotaleite]

@gracemukendi

at least for your case, you can probably use something like this directly

bootstrap_servers = split(",",aws_msk_cluster.msk-cluster.bootstrap_brokers_sasl_scram)

But i do agree that the bootstrap_servers could be a little more flexible and avoid the need for split()

[antoinedeschenes]

Any chances https://github.com/Mongey/terraform-provider-kafka/pull/131 might work now?

We have a use case where we want to destroy resources without the original terraform module/variables available, as this destruction happens when the module is removed from configuration.

The Terraform state doesn't keep provider variables, so Terraform attempts to setup the provider using default settings, but fails at The argument "bootstrap_servers" is required, but was not set.